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Abstract 

As  the  nature  of  warfare  has  shifted  from  a  conventional  approach  to  more 
guerilla  type  warfare,  intelligence  has  become  more  important  than  at  any  other  time  in 
the  history  of  the  United  States  Military.  With  the  stochastic  nature  of  intelligence 
gathering,  it  is  almost  impossible  to  know  with  any  degree  of  certainty  where  and  when 
the  next  piece  of  information  that  could  possibly  change  the  course  of  the  battle  or  war 
will  be  obtained.  US  intelligence  gathering  assets  have  long  been  plagued  with  using 
useless  measures  of  performance  rather  than  measures  of  effectiveness  to  determine  their 
worth.  This  research  uses  a  value  focused  thinking  approach  to  determine  the 
effectiveness  of  a  specific  capability  or  asset.  Specifically,  it  looks  at  Ground  Moving 
Target  Indicator  onboard  the  E-8C  Joint  Surveillance  Target  Attack  Radar  System.  This 
research  attempts  to  provide  a  model  to  a  decision  maker  so  he  or  she  will  know  in 
advance  the  approximate  value  of  information  they  will  receive  from  a  particular  asset  or 
capability  before  the  asset  is  ever  deployed  into  the  area  of  responsibility. 
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Chapter  1.  Introduction 


Measure  of  Effectiveness  for  JSTARS  Ground  Moving  Target  Indicator:  A  Value 

Focused  Thinking  Approach 


1.1  Background 

"Electronic  intelligence,  valuable  though  it  is  in  its  own  way,  serves  to  augment  the 
daunting  volume  of  information  which  is  directed  at  headquarters  from  satellite  and 
aerial  reconnaissance,  intelligence-gathering  ships,  optical  obserx’ation,  Special  Forces, 
armored  reconnaissance  teams,  and  the  interrogation  of  prisoners.  Nowadays  the 
commander  is  confronted  with  too  much  information,  rather  than  too  little,  and  it  is  his 
informed  judgment  which  ultimately  decides  what  is  relevant  and  important. " 
(Fcirringdon) 

The  Department  of  Defense  (DoD)  has  an  enormous  amount  of  assets  that  are 
dedicated  to  the  intelligence  gathering  process.  From  satellites  out  in  space  to 
remotely  piloted  vehicles,  there  is  a  great  amount  of  the  DoD  budget  dedicated  to 
gathering  intelligence.  In  today’s  war  environments  where  the  fighting  is  mostly 
unconventional,  we  depend  on  our  intelligence  gathering  platforms  more  than  ever  to 
provide  timely  and  accurate  information.  A  problem  that  has  long  since  plagued  the 
intelligence  gathering  systems  is  differentiating  between  measures  of  performance 
and  measures  of  effectiveness.  Measures  of  Effectiveness  (MOEs)  are  quantitative 
measures  that  give  some  insight  into  how  effectively  a  unit  is  performing,  and 
Measures  of  Performance  (MOPs)  describes  how  well  a  system  utilizes  resources.  For 
some  of  the  systems  it  is  as  simple  as  taking  a  picture  and  then  evaluating  that  picture 
to  determine  if  it  obtained  the  information  you  required.  In  this  example  it  would  be 
somewhat  easy  to  determine  some  measure  of  effectiveness  and  then  build  a  model  to 
determine  if  the  picture  that  was  taken  meets  some  threshold  of  satisfaction. 
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However,  there  are  other  intelligence  gathering  capabilities  that  are  not  as  easy  to 
determine  a  true  measure  of  effectiveness.  For  example,  how  many  hours  does  the 
RC-135  (Rivet  Joint)  have  to  orbit  using  its  COMINT  or  ELINT  capability  to  be 
considered  effective?  How  long  does  the  E-8  JSTARS  (Joint  Surveillance  Target 
Attack  Radar  System)  have  to  orbit  to  be  considered  effective?  These  are  difficult 
questions  to  answer  as  who  knows  when  either  of  these  aircraft  will  gather  that  one 
piece  of  intelligence  that  could  possibly  change  the  course  of  the  war.  A  piece  of 
information  that  could  be  key  in  changing  the  course  of  the  battle  could  come  on  the 
first  intercepted  transmission  of  the  night,  the  last  before  they  go  off  station,  or  not  at 
all.  Because  of  the  sheer  uncertainty  of  the  intelligence  gathering  process,  DoD  has 
failed  to  place  true  measures  of  effectiveness  on  many  of  the  platforms  and/or  their 
capabilities.  Instead,  in  the  absence  of  true  measures  of  effectiveness,  the 
effectiveness  of  these  platforms  is  measured  by  a  measure  of  performance.  It’s  much 
easier  and  less  complicated  to  assign  a  platform  to  go  orbit  for  some  duration  of  time 
and  then  measure  its  effectiveness  by  how  long  during  that  assigned  station  time  they 
were  actually  on  station.  Measurements  such  as  the  previous  example  can  lead  to 
very  high  levels  of  effectiveness  and  look  like  a  very  attractive  asset  when  doing  an 
assessment  of  the  best  DoD  intelligence  assets.  The  problem  however  with  these 
types  of  measurements  is,  if  a  detailed  analysis  where  conducted  you  may  find  that 
the  true  effectiveness  of  these  assets  could  be  extremely  low  to  almost  zero. 

1.2  Problem  Statement 

In  sum,  the  security  challenges  we  now  face  and  will  in  the  future  have  changed,  and  our 

thinking  must  likewise  change.  The  old  paradigm  of  looking  at  a  potential  conflict  as 
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either  regular  or  irregular  war,  conventional  or  unconventional,  high-end  or  low-end  is 
no  longer  relevant.  And  as  a  result,  the  Defense  Department  needs  to  think  about  and 
prepare  for  war  in  a  profoundly  different  way  than  what  we  have  been  accustomed  to 
throughout  the  better  part  of  the  last  century.  In  truth,  preparing  for  conflict  in  the  21st 
century  means  investing  in  truly  new  concepts  and  new  technologies  ( Gates,  US 
Department  of  Defense,  2010). 

It  means  taking  into  account  cdl  the  assets  and  capabilities  we  can  bring  to  the  fight.  It 
means  measuring  those  capabilities  against  real  threats  posed  by  real  world  adversaries 
with  reed  limitations,  not  threats  conjured  up  from  enemies  with  unlimited  time,  unlimited 
resources  and  unlimited  technological  acumen.  (Gates,  US  Department  of  Defense,  2009) 


In  today’s  budget  sensitive  economy  there  is  a  battle  waged  over  every  dollar  in 
the  DoD  budget.  No  more  are  the  days  of  limitless  and  unchecked  military  spending. 
A  growing  chorus  of  politicians  and  citizens  are  calling  for  defense  spending  to  be 
scrutinized  as  much  as  any  other  federal  program  when  it  comes  time  to  tighten  the 
nation's  fiscal  belt.  At  $689  billion  this  year,  defense  spending  accounts  for  about 
20%  of  the  entire  federal  budget  and  it  makes  up  50%  of  the  so-called  discretionary 
budget,  which  pays  for  everything  but  entitlement  programs  and  interest  on  the  debt. 
(Sahadi,  2010).  The  DoD  is  now  under  scrutiny  to  find  ways  to  cut  useless  and 
redundant  equipment  and  systems.  They  have  to  justify  ever  piece  of  equipment  it 
contends  it  requires  to  maintain  the  safety  and  security  of  the  nation.  With  such  tight 
constraints,  every  asset  has  to  prove  its  worthiness  or  face  possible  crippling  budget 
cuts. 


The  “shock  and  awe”  strategy  is  a  very  popular  one  among  senior  officials,  but 
the  wars  in  Afghanistan  and  Iraqi  show  clearly  that  massive  applications  of  force 
have  done  little  more  than  kill  the  innocent  and  enrage  their  survivors  (Arquilla, 
2010).  As  the  nature  of  warfare  has  changed  so  has  the  systems  needed  to  fight 
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successfully  and  win  the  changing  and  dynamic  battlefield.  The  paradigm  of  always 
“fighting  the  last  war”  has  been  scrutinized  severely  and  we  can  only  afford  to  keep 
systems  that  will  allow  us  to  win  the  next  war.  Keeping  and  maintaining  systems  in 
the  DoD  arsenal  because  they  have  always  been  a  part,  no  longer  meets  fiscal 
constraints.  It  has  become  imperative  that  every  weapon  system  have  measures  of 
effectiveness  that  show  that  they  can  complete  their  mission  in  a  manner  that  is  cost 
effective  to  the  tax-payer,  and  shows  that  it’s  contributing  to  the  overall  mission  each 
and  every  time  the  asset  is  employed. 

1.3  Research  Objective 

The  value  focused  thinking  (VFT)  process  has  been  used  in  several  different 
applications  over  the  years  from  determining  force  protection  initiatives  (Jurk,  2002) 
to  determining  security  solutions  for  Homeland  Security  (Pruitt,  2003).  This  is  not  a 
new  methodology.  The  goal  of  this  research  is  to  use  the  VFT  methodology  to  model 
the  collection  of  any  intelligence  gathering  asset  by  developing  a  model  that  can 
consistently  and  accurately  measure  how  effective  an  asset  or  capability  will  be  in 
any  given  scenario.  This  will  enable  planners,  collection  managers,  and  flight  crews 
to  have  a  much  better  idea  of  the  value  of  information  they  will  receive  prior  to 
collection  deck  completion  or  pulling  back  on  the  yolk.  This  research  will  force  all 
involved  to  look  at  the  constraints  of  the  mission  and  determine  if  there  are  any  things 
they  can  change  prior  to  the  mission  to  improve  the  value  of  information  they  will 
receive.  This  research  will  also  force  commanders  and  decision  makers  to  reanalyze 
whether  they  want  to  spend  thousands  of  pounds  of  fuel,  hundreds  of  man  hours,  and 
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other  coveted  Air  Force  resources  to  fly  missions  that  will  potentially  result  in  very 


low  information  value. 

The  remainder  of  this  document  will  consist  of  a  literature  review  section,  a 
methodology  section,  a  results  and  analysis  section  and  finally  conclusions  and 
recommendations.  The  literature  review  section  will  discuss  all  information  that  is 
pertinent  to  intelligence,  JSTARS,  decision  analysis  and  theory  of  measurement.  The 
methodology  section  will  discuss  in  detail  the  10-step  value-focused  thinking 
methodology.  Results  and  analysis  will  discuss  how  the  model  was  created  and  what 
the  results  of  the  analysis  of  the  output  were.  Finally,  in  section  5  the  conclusions  of 
the  study  will  be  documented  along  with  some  recommendations  on  how  to  proceed 
in  the  future  will  be  presented. 

Chapter  2.  Literature  Review 


2.1  Introduction 

To  accurately  model  any  process  you  must  first  understand  what  research  has 
already  taken  place  in  the  area  of  interest.  Also  there  needs  to  be  an  understanding  of 
other  research  and  methodologies  that  do  not  directly  relate  to  the  area  of  interest,  but 
could  possibly  be  adapted  to  the  issue  or  problem.  This  chapter  provides  background  on 
the  4  main  areas  in  which  this  research  needs  to  be  effective.  Section  2  focuses  on  the 
intelligence  process,  the  how  and  why  we  acquire  and  need  certain  types  of  intelligence. 
Section  3  will  focus  mainly  on  the  E8-C  Joint  Surveillance  Target  Attack  Radar  System 
and  the  major  capability  it  features  which  is  Ground  Moving  Target  Indicator  (GMTI). 
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Section  4  will  concentrate  on  the  Theory  of  Measures  of  Effectiveness,  the  root  on  which 


we  determine  how  well  a  system  is  or  is  not  performing.  Finally,  section  5  will  give  a 
brief  overview  of  decision  analysis  and  the  value  focused  thinking  methodology  which 
will  be  applied  and  explained  in  chapters  3  and  4  of  this  thesis  in  much  greater  detail. 


2.2.  Intelligence  and  Intelligence  Capabilities 

The  purpose  of  collecting  intelligence  is  to  inform  the  commander,  identify/define 
objectives,  support  planning  and  execution,  counter  the  adversary,  support  friendly 
deception,  and  to  assess  the  effects  of  the  operation  (Defense,  Joint  Intelligence,  JP  2.0, 
2007).  Before  you  can  do  any  of  these  things  you  have  to  understand  what  the  true 
meaning  of  the  word  intelligence  is.  As  with  any  popular  word  there  are  multiple  ways  in 
which  intelligence  is  defined.  Some  of  the  more  simplified  definitions  located  in  the 
dictionary  state  intelligence  is  the  ability  to  learn  or  understand  or  deal  with  new  or  trying 
situations.  It  is  also  defined  as  the  ability  to  apply  knowledge  to  manipulate  one’s 
environment  or  to  think  abstractly  as  measured  by  objective  criteria.  The  final  Webster’s 
definition  of  intelligence  is  information  concerning  an  enemy  or  a  possible  enemy  area 
(Merrian-Webster,  201 1).  Among  the  three  different  versions,  a  more  hybrid  approach  is 
most  appropriate  for  a  military  organization  as  it  is  concerned  with  the  “ability  to  learn 
and  understand”,  they  are  also  interested  in  “knowledge”,  and  finally  they  are  concerned 
with  information  as  it  deals  with  their  “enemies.”  In  laymen’s  terms  they  need  to  have 
the  ability  to  gather  knowledge  on  our  enemy  so  that  we  have  the  ability  to  learn  and 
understand  them  and  ultimately  defeat  them.  In  the  joint  environment,  intelligence  is 


6 


defined  as  the  product  resulting  from  the  collection,  processing,  integration,  evaluation, 
analysis  and  interpretation  of  available  information  concerning  foreign  nations,  hostiles 
or  potentially  hostile  forces  or  elements,  or  areas  of  actual  or  potential  operations 
(Defense,  Joint  Intelligence,  JP  2.0,  2007).  This  definition  goes  far  more  in-depth  on  the 
processing,  evaluation  and  analysis  of  the  data.  All  types  of  information  can  be  collected, 
but  if  there  is  no  accurate  well  thought  out  procedure  to  exploit  the  data  then  the  eight 
attributes  of  intelligence  excellence  that  are  located  in  appendix  E  will  not  be  met. 

A  derivative  of  the  intelligence  process  is  ISR  or  Intelligence,  Surveillance,  and 
Reconnaissance.  The  goal  of  the  ISR  process  is  to  provide  accurate,  relevant,  and  timely 
intelligence  to  decision  makers  (AFDD,  2007).  ISR  plays  a  crucial  role  in  achieving 
decision  superiority  as  it  gives  commanders  a  competitive  advantage  by  ensuring  he  and 
his  troops  have  the  situational  awareness  to  make  better  informed  decisions.  Of  course 
ISR  is  broken  down  into  the  three  components,  of  which  intelligence  has  already  been 
discussed.  However  surveillance  is  defined  as  “the  systematic  observation  of  aerospace, 
surface  or  subsurface  areas,  places,  persons,  or  things,  by  visual,  aura,  electronic, 
photographic  or  other  means.”  Reconnaissance  is  defined  as  “a  mission  undertaken,  by 
visual  observation  or  other  detection  methods,  information  about  the  activities  and 
resources  of  an  enemy  or  potential  enemy”  (AFDD,  2007).  The  information  derived  from 
surveillance  and  reconnaissance  is  exploited  and  analyzed  and  turned  into  intelligence. 
The  key  principles  of  ISR  are  that  it  must  be  integrated,  accurate,  relevant,  timely,  fused, 
accessible,  secure,  survivable,  sustainable  and  deployable.  ISR  is  undoubtedly  one  of  the 
most  important  aspects  of  the  intelligence  process  and  cannot  be  done  effectively  without 
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some  of  the  major  ISR  assets  such  as  JSTARS,  Rivet  Joint,  Global  Hawk  and  other 
airborne  and  space  assets. 

2.3  The  History  of  Ground  Moving  Target  Indicator  (GMTI)  and  Joint  Surveillance 
Target  Attack  Radar  System  (JSTARS) 

The  development  of  GMTI  dates  back  to  the  Arab-Israeli  War  in  1973.  During  a 
fact-finding  tour,  the  US  Army  noted  Arab  and  Israeli  forces  had  lost  more  tanks  in  a  six- 
day  conflict  than  they  had  deployed  in  the  entire  European  theater  at  the  time  (Dunn, 
Bingham,  &  Fowler,  2004).  Noting  the  lethality  of  the  new  battlefield,  General  William 
DePuy,  Commander  of  the  Army’s  Training  and  Doctrine  Command  (TRADOC), 
recognized  “field  commanders  would  have  to  know  the  enemy’s  situation  beyond  the 
front  line”,  to  include  his  successive  echelons,  artillery,  support  troops,  headquarters,  and 
possible  courses  of  action.  In  1982  the  new  TRADOC  commander,  General  Donn  Starry, 
expanded  the  doctrine  to  include  Soviet  second-echelon  forces  which  focused  on  the  need 
to  synchronize  air  and  ground  power  at  the  operational  level.  Recognizing  the  need  for  a 
collaborative  effort,  the  Army  and  Air  Force  entered  into  a  joint  agreement  in  1983  to 
explore  31  specific  initiatives  supporting  air  and  ground  operations  (Dunn,  Bingham,  & 
Fowler,  2004). 

The  Army’s  Stand-Off  Target  Acquisition  System  (SOT AS)  along  with  the  Air 
Force’s  Assault  Breaker/Pave  Mover  were  the  precursors  to  the  modern  GMTI  radar.  The 
SOTAS  was  mounted  onboard  a  helicopter  and  gained  commanders  support  when  they 
realized  the  value  of  seeing  the  opposing  forces  movement.  Although  there  was  strong 
support  from  field  commanders  for  this  program,  due  to  cost  overruns,  the  program  was 
cancelled  in  1980.  While  the  Army  was  doing  their  research,  the  AF  was  also  conducting 


significant  research  on  adding  the  GMTI  capability  to  fast-moving  aircraft.  In  1976  the 
Defense  Science  Board  conducted  a  study  that  proposed  an  alternative  to  countering  the 
Warsaw  Pact  by  locating  and  attacking  the  second  and  third  echelon  forces  with  air  and 
ground  missiles  (Dunn,  Bingham,  &  Fowler,  2004).  In  support  of  this  effort 
Grumman/Norden  changed  the  emphasis  of  its  Radar  Guided  Weapon  System  to  a  side 
looking  GMTI  system  which  gave  them  a  head  start  in  the  Pave  Mover.  The  Pave  Mover 
system  was  initially  installed  on  the  F-l  1 1.  The  radar  had  the  ability  to  switch  rapidly 
from  GMTI  to  Synthetic  Aperture  Radar  (SAR)  mode  giving  high  resolution  images  of 
areas  of  interest.  It  became  obvious  that  neither  Congress  nor  the  Office  of  the  Secretary 
of  Defense  were  going  to  fund  two  separate  GMTI  programs  and  urged  the  forces  to 
combine  their  efforts.  Selecting  one  aircraft  to  satisfy  both  services  requirements  was 
quite  difficult  because  the  Army  wanted  a  dedicated  intelligence  Surveillance  and 
Reconnaissance  (ISR)  asset,  where  as  the  AF  wanted  a  Battle  Management  asset  to  guide 
aircraft  and  missiles  and  also  provide  ISR.  Eventually  both  service  chiefs  signed  a 
Memorandum  of  agreement  that  JSTARS  would  be  the  aircraft  and  the  prioritization  of 
its  missions  would  be  equitable  easing  Army  tensions  since  the  AF  would  be  responsible 
for  operating  the  aircraft.  In  1985  Grumman/Norden  was  awarded  the  contract  to  build 
the  10  aircraft  with  4  additional  developmental  aircraft. 

JSTARS  is  a  Boeing  707  aircraft  that  has  several  different  missions  which  include 
Air  Interdiction,  Airborne  Battlefield  Command  and  Control  Center,  Close  Air  Support, 
Command  and  Control  and  ISR  (Albers,  2001).  The  basic  crew  consists  of  18  people 
which  include  a  Pilot,  Co-Pilot,  Navigator,  Engineer,  Mission  Crew  Commander(MCC), 
Deputy  Mission  Crew  Commander  (DMCC,  Army),  Senior  Director  (SD),  Airborne 
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Weapons  Officer  (AWO,  2),  Senior  Surveillance  Manager  (SSM),  Airborne  Operation 
Technicians  (AOT,  2),  Airborne  Target  Surveillance  Supervisors  (ATSS,  Army,  2), 
Airborne  Intelligence  Officer/Technician  (AIO/T),  Communication  Systems  Technician 
(2)  and  an  Airborne  Radar  Technician  (2)  (Vol  3,  2009).  A  more  detailed  description  of 
the  different  jobs  onboard  the  aircraft  can  be  found  in  the  appendix  of  this  document. 
Figure  1  shows  the  hierarchy  onboard  the  aircraft  and  how  the  positions  interact  with  one 
another.  The  aircraft  has  1 8  workstations  in  the  back  of  the  aircraft,  but  not  all  are  used 
for  ISR  and  battle  management.  Four  of  the  consoles  are  obligated  to  airborne  system 
maintenance  as  the  CST’s  and  ART’s  use  these  consoles.  One  more  is  used  for  the 
navigator  to  ensure  they  have  total  situational  awareness  of  the  actions  that  are  taking 
place  on  both  ends  of  the  aircraft. 


Figure  1:  The  JSTARS  Crew  Composition 


10 


The  aircraft  has  a  24-foot,  canoe  shaped  side-looking  phased  array  radar  in  a 
dome  underneath  the  aircraft.  The  crew  conducts  its  operations  by  establishing  a  figure 
eight  or  race  track  orbit  which  is  at  least  50  kilometers  away  and  no  further  than  250 
kilometers  away  from  the  area  of  interest  (Albers,  2001).  The  radar  has  two  modes  of 
operation.  Moving  Target  Indicator  (MTI)  and  Synthetic  Aperture  Radar  (SAR)  and  has  a 
120  degree  field  of  view  that  can  detect  targets  up  to  125  knots.  The  radar  detects  targets 
using  a  Doppler  shift  or  a  double  Doppler  shift.  Track  vehicles  such  as  tanks  are  detected 
because  the  tracks  on  the  vehicle  are  typically  moving  twice  as  fast  as  the  vehicle  itself. 
The  tracks,  or  their  direction  of  movement,  are  displayed  on  the  operator’s  console. 
Magenta  dots  represent  wheeled  vehicles  or  Doppler  shift  and  yellow  dots  represent  track 
vechiles  or  double  Doppler  shift  (Albers,  2001).  In  theory  an  operator  should  be  able  to 
tell  what  type  of  vehicle  it  is  based  on  the  color  of  the  tracks,  but  in  practice  it  has  been 
shown  that  this  is  not  a  reliable  way  to  identify  the  targets  and  there  needs  to  be  some 
type  of  cross-cue  from  another  asset  with  video  or  eyes  on  the  target  to  insure  accuracy. 

Prior  to  mission  planning  or  during  the  mission,  radar  service  request  (RSRs)  are 
received  from  supported  agencies  and  establish  the  priorities  for  the  radar.  The  radar 
sweeps  the  requested  areas  and  provides  the  information  to  the  on-board  operators.  The 
frequency  or  revisit  rate  in  which  these  are  looked  at  by  the  radar  is  based  on  the  priority 
of  the  job.  The  radar  has  a  limited  amount  of  RSRs  that  it  can  process  at  any  given  time. 
The  more  RSRs  that  are  requested  affects  the  timeline  of  the  radar  which  results  in  lower 
priority  jobs  not  being  processed  in  accordance  with  the  agreed  upon  timeline. 

There  are  at  least  five  different  RSRs  that  the  radar  can  provide  in  the  MTI  mode.  The 
first  is  the  Ground  Reference  Coverage  Area  (GRCA)  which  is  a  wide  area  surveillance 
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(WAS)  which  has  low  resolution  and  low  priority.  The  GRCA  is  the  area  in  which  the 
radar  will  attempt  to  continually  keep  in  view  no  matter  its  position  in  the  orbit  and  is 
generally  an  area  of  160  x  180  kilometers.  A  standard  revisit  rate  on  the  GRAC  is  sixty 
seconds,  which  means  the  radar  will  attempt  to  give  an  update  of  the  MTI  picture  of  the 
mission  area  every  sixty  seconds.  The  next  one  is  the  Radar  Reference  Coverage  Area 
(RRCA)  which  is  another  low  resolution  and  low  priority  job.  The  RRCA  is  fixed 
azimuth  ninety  degrees  off  the  wing  and  does  not  have  a  defined  search  area.  This  mode 
is  normally  used  enroute  to  the  Area  of  Responsibility  (AOR)  to  check  the  accuracy  of 
the  radar.  The  Sector  Search  (SS)  is  an  RSR  that  is  smaller  in  size  than  the  GRCA  and 
provides  a  higher  resolution,  higher  revisit  rate  and  is  a  higher  priority  job.  The  SS 
provides  more  accurate  and  timelier  MTI  data  than  the  GRCA.  The  Attack  Control  (AC) 
is  a  high  resolution,  high  priority  RSR  that  has  an  even  higher  revisit  rate  than  the  SS. 
The  AC  is  usually  smaller  than  the  SS  and  is  the  RSR  that  is  most  commonly  used  for 
targeting.  The  final  MTI  RSR  is  the  Attack  Planning  (AP)  which  has  high  resolution,  but 
its  priority  and  revisit  rate  are  lower  than  the  AC.  Since  this  mode  is  very  similar  to  the 
AC  it  is  rarely  used  during  an  operational  mission  (Albers,  2001). 

The  second  mode  the  radar  is  operated  in  is  SAR.  In  this  mode  the  radar  focuses 
on  a  specific  area  and  creates  a  radar  image  of  the  area.  SARs  are  high  resolution  RSRs 
and  they  use  a  much  more  of  the  radar  time  than  any  other  RSR.  SARs  are  also  the 
highest  priority  RSR  and  once  approved  they  are  completed  before  any  other  job  can  be 
done.  SARs  can  also  be  taken  in  the  Fixed  Target  Indicator  (FTI)  Mode.  When  taken  in 
this  mode  red  dots  are  overlaid  on  the  SAR  picture  signifying  the  area  of  the  greatest 
returns.  In  general,  SARs  are  used  for  battle  damage  assessments  and  in  the  FTI  mode  to 
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indentify  buildings,  stationary  vehicles,  or  assembly  areas.  The  field  of  view  in  Figure  2 
shows  the  special  relationships  between  the  aircraft  and  the  different  Radar  Service 
Request. 


Figure  2:  JSTARS  Field  of  View 

The  JSTARS  was  given  an  opportunity  to  show  the  value  of  GMTI  even  before  it 
was  an  operational  aircraft.  Army  Lieutenant  General  Fred  Franks  had  been  quite 
impressed  with  the  capability  of  GMTI  and  convinced  Army  Senior  leaders  to  insist  on 
deploying  the  aircraft  to  the  Gulf  War  while  it  was  still  in  its  developmental  phase  (Dunn, 
Bingham,  &  Fowler,  2004).  With  no  operational  experience  and  with  civilian  and 
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military  crews  receiving  on-the-job  training,  the  crews  quickly  began  to  exploit  the 
GMTI  capabilities.  The  JSTARS  crews  were  the  first  to  locate  advancing  Iraqi  forces  that 
were  moving  into  Saudi  Arabia  during  the  Battle  of  Khafji  (Clevenger,  1996).  GMTI 
played  a  role  in  assuring  coalition  leaders  that  the  movement  was  indeed  an  attack  and 
not  military  deception.  The  JSTARS  proved  GMTI  was  a  unique  and  valuable  capability 
that  had  changed  the  war.  Brigadier  General  John  F.  Stewart,  the  Army’s  senior 
intelligence  officer  at  the  time  stated  “the  JSTARS  was  the  single  most  valuable 
intelligence  and  targeting  collection  system  in  Desert  Storm”  (Stewart,  1991). 

During  Operation  Allied  Force  the  JSTARS  was  called  upon  again  to  monitor  the 
ground  movement  of  the  enemy  from  above.  Unlike  the  wide  open  desert  of  Saudi 
Arabia,  Kosovo  terrain  was  rugged  and  full  of  foliage  which  increased  the  amount  of 
radar  screening  dramatically.  Another  factor  that  limited  the  effectiveness  of  GMTI  was 
the  fact  there  were  very  few  friendly  ground  troops,  which  allowed  the  Serb  forces  to 
disperse  and  escape  from  being  targeted  and  attacked.  The  distance  they  had  to  fly  to 
their  orbits  and  the  low  number  of  aircraft  (4)  did  not  allow  them  to  fly  persistent  24  hour 
coverage  which  allowed  Serb  forces  several  opportunities  to  move  without  being 
detected.  Finally,  the  orbits  that  they  were  assigned  were  not  optimized  to  prevent  radar 
screening.  While  some  of  these  issues  were  mitigated  once  liaison  officers  were 
deployed  to  the  Air  Operations  Center,  it  was  clear  that  GMTI  did  not  provide  the  earth 
shattering  intelligence  it  did  during  Desert  Storm  (Dunn,  Bingham,  &  Fowler,  2004). 

JSTARS  was  once  again  called  to  duty  in  Operation  Enduring  Freedom.  This 
environment  also  provided  a  myriad  of  challenges  to  JSTARS  and  the  use  of  GMTI. 
Radar  screening  was  a  huge  issue  as  the  terrain  in  the  AOR  is  quite  mountainous; 
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however  GMTI  was  more  effective  when  the  terrain  channeled  movement.  While  the 
vehicle  movement  was  not  as  robust  as  in  past  scenarios,  the  fact  that  there  was  a  much 
larger  contingent  of  ground  forces  enhanced  the  effectiveness  of  GMTI.  Orbit  and 
altitude  were  also  major  issues  as  the  aircraft  was  placed  in  orbits  that  were  ineffective  or 
the  aircrafts  could  not  reach  altitudes  that  would  decrease  the  radar  screening.  Another 
issue  that  was  discovered  was  the  time  period  that  JSTARS  arrived  in  the  theater.  Since 
JSTARS  arrived  after  combat  operations  had  already  begun,  much  of  the  vehicular 
movement  of  the  Taliban  and  al  Qaeda  was  no  longer  occurring  and  valuable  targeting 
opportunities  for  GMTI  had  been  missed  (Dunn,  Bingham,  &  Fowler,  2004).  One 
significant  break  though  during  this  conflict  was  the  cross  cueing  with  remotely  piloted 
vehicles  which  proved  to  enhance  the  intelligence  capability. 

During  Operation  Iraqi  Freedom,  the  United  States  Military  had  the  advantage  of 
the  lessons  learned  from  recent  wars  and  used  that  knowledge  to  do  things  much  smarter. 
For  the  first  time,  several  aircraft  were  to  provide  coverage  and  to  collect  baseline  data 
before  the  conflict  began.  Once  the  conflict  began,  Iraqi  forces  had  a  dilemma.  They 
could  remain  immobile  and  be  easily  defeated  or  they  could  move  and  risk  being  seen  by 
GMTI  and  targeted  by  coalition  forces.  GMTI  was  also  used  in  a  new  way  to  provide 
protective  watch  of  coalition  supply  lines  which  allowed  the  forces  to  respond  to  Iraqi 
forces  of  significant  size. 

The  JSTARS  weapon  system  has  had  some  great  success  and  some  extreme 
failures.  Throughout  these  different  conflicts  however,  a  template  of  how  to  best  employ 
the  weapons  system  to  maximize  its  effectiveness  has  emerged.  Using  this  historical  data 
along  with  knowledge  of  current  employment  strategies,  this  research  develops  measures 
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of  effectiveness  used  in  a  model  to  determine  what  scenarios  maximize  the  effectiveness 


of  this  asset  and  capability. 


2.4  Theory  of  Effectiveness  Measurements 

“ Don't  lower  your  expectations  to  meet  your  performance.  Raise  your  level  of 
performance  to  meet  your  expectations.  Expect  the  best  of  yourself  and  then  do  what  is 
necessary  to  make  it  a  reality.  ’’  (Marston,  2009) 

Measurement  is  an  integral  part  of  our  daily  lives.  Measurement  is  closely  aligned 
with  physical  science  and  is  deterministic  in  nature.  Unfortunately  some  fields  such  as  the 
social  and  behavioral  sciences  have  events  and  processes  that  are  difficult  to  understand 
and  very  difficult  to  measure.  Military  intelligence  gathering  is  another  example  where  it 
is  extremely  difficult  to  measure  effectiveness  because  of  the  dynamics  and 
unpredictability  of  when,  where  and  even  how  it  is  obtained.  The  challenge  in  gathering 
intelligence  is  the  nature  of  intelligence  is  stochastic  and  dynamic  and  really  does  not 
exhibit  any  deterministic  traits. 

Effectiveness  measures  provide  the  critical  link  between  strategy  and  execution, 
essentially  translating  strategy  into  reality  (Melenyk,  Stewart,  &  Swink,  2004).  Measures 
of  effectiveness  directly  influence  how  decision  makers  assess  the  impact  of  deliberate 
actions  and  affect  critical  issues  such  as  resource  allocation  as  well  as  whether  to 
maintain  or  change  the  existing  strategy  (Gartner,  1997).  The  lack  of  a  foundation  and 
framework  can  lead  to  erroneous  measures  that  really  don’t  accurately  measure  what  they 
are  intended  to  measure.  Measurements  in  military  environments  can  contain  error 
yielding  uncertainty  concerning  the  true  state  of  the  system  resulting  from  deliberate 
actions. 
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Measurement  is  needed  to  capture  information  about  the  system  through  their 
attributes  which  can  be  directly  or  indirectly  observable  (Bullock,  2006).  Measurement  is 
an  abstraction  because  it  does  not  directly  measure  the  system,  but  only  addresses  the 
attributes  about  the  system  (Pfanzagl,  1971).  In  other  words  measurement  can  be  thought 
of  as  a  process  that  assigns  symbols  to  attributes  that  reflect  the  underlying  nature  of  the 
system  (Bullock,  2006).  However,  attribute  selection  is  crucial  since  the  validity  of  the 
system  measurement  is  influenced  by  the  number  of  attributes  used  in  the  measurement. 
While  a  small  number  of  attributes  can  simplify  the  measurement  process,  too  few  can 
lead  to  poor  or  misleading  insights  about  a  system. 

Once  attributes  are  identified,  observations  or  data  collection  on  the  system  can 
begin.  There  may  be  several  different  ways  to  measure,  but  whatever  measurement  is 
used  it  is  just  a  raw  symbol  derived  from  the  observation  while  an  indicator,  or  index,  is  a 
measure  for  a  complex  attribute  (Bullock,  2006).  Good  measures  are  generally 
characterized  as  being  valid,  reliable,  and  have  some  type  of  amplitude.  The  validity  of 
any  measure  is  affected  by  its  attribute,  because  validity  characterizes  how  well  a 
measure  reflects  the  system  attributes  it  was  supposed  to  represent.  Reliability  addresses 
the  consistency  or  repeatability  of  the  measure,  and  amplitude  demonstrates  how  well  the 
measure  represents  high  order  constructs  and  complex  attributes  (Geisler,  2000). 

Typically  when  something  is  measured,  it  is  done  with  some  type  of  instrument. 
That  instrument  can  be  as  simple  as  a  ruler  or  as  complicated  as  a  mathematical  model 
(Bullock,  2006).  Regardless  of  the  form,  the  underlying  relationship  between  the 
instrument  and  the  attribute  being  measured  must  be  the  same.  The  problem  is  that  scales 
themselves  can  be  a  source  of  error,  since  most  measures  have  some  type  of  inherent 
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error.  The  primary  source  of  measurement  error  comes  from  random,  systemic,  and 
observational  error.  Random  error  is  the  stochastic  variation  that  can  be  generated  from 
anywhere.  Systemic  error  is  derived  from  the  construction  of  the  measure  or  definition  of 
the  measurement  bias.  Observational  error  is  the  oversight  of  key  systems  attributes 
requiring  measurement  or  using  the  wrong  measures  for  the  indentified  system  attributes 
(Bullock,  2006).  Error  is  inescapable,  but  through  statistics  we  can  make  inferences  on 
the  data  that  is  either  input  or  output. 

To  measure  a  system  properly,  it  is  imperative  that  something  is  known  about  the 
system.  Unfortunately,  the  reason  measurement  is  required  is  because  there  is  a  need  to 
get  a  better  understanding  of  the  system  (Geisler,  2000).  For  complex  systems  the 
attributes  of  the  system  may  be  unknown  and  require  a  proxy  or  indirect  method  of 
measurement  such  as  a  mathematical  model  or  some  type  of  approximation.  There  is 
really  no  easy  way  to  derive  the  proxies  of  the  systems  and  usually  requires  breaking 
complex  systems  down  into  understandable,  measurable  components. 

The  most  widely  accepted  form  of  a  measure  is  the  representational  view  which  is 
built  upon  their  representation,  uniqueness,  and  meaningfulness.  For  a  system  to  be 
measureable,  it  must  somehow  map  a  formal  domain  into  an  empirical  domain.  Simply 
stated,  there  must  be  some  rational  way  to  turn  the  attribute  of  the  systems  into  an 
applicable  measure. 

There  are  at  least  nine  different  scale  types,  but  the  most  common  are  Nominal, 
Ordinal,  Interval,  Ration  and  Absolute.  Nominal  scales  only  have  equivalence  meaning, 
where  ordinal  scales  have  both  equivalence  and  rank  meaning.  Interval  scales  have  both 
equivalence  and  rank  meaning,  but  also  have  some  meaning  in  the  intervals  between 
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values.  Ratio  scales  have  all  of  the  preceding  meanings  but  add  a  ratio  value  meaning  and 
absolute  scales  are  ratios  with  no  units  attached,  but  are  often  interpreted  as  a 
measurement  by  counting.  Each  higher  level  scale  can  always  be  converted  to  a  lower 
level  scale,  but  a  lower  level  scale  cannot  be  converted  in  to  a  higher  level  scale. 

To  create  good  measures  you  must  first  have  a  measurement  plan.  The 
measurement  plan  should  address  the  information  to  be  derived  from  the  measurement 
activity  (Park,  Goethert,  &  Florae,  1996)  and  how  the  system  will  be  measured.  This 
should  include  how  measures  will  be  determined  and  how  measures  will  be  collected,  as 
well  as  allocation  of  resources  for  the  measurement  activities  to  include  training  and 
tools.  The  plan  should  be  a  living  document  which  serves  to  guide  the  measurement 
process,  document  the  process,  and  provide  an  audit  trail  for  the  system  measurement 
process  (Sproles,  1997).  A  good  measure  can  also  yield  information  on  when  and  why  a 
system  is  deviating  from  its  normal  behavior,  but  in  order  to  receive  maximum  benefit 
the  measurement  must  be  an  explicit  and  objective  activity. 

Measures  of  effectiveness  (MOE),  measures  of  performance  (MOP),  and 
measures  of  outcome  (MOO)  are  the  three  types  of  measures  typically  used  to  measure  a 
system.  MOEs  provide  insights  on  how  well  a  system  tracks  against  its  purpose  and 
MOPs  describe  how  well  a  systems  utilizes  its  resources.  In  other  words,  MOEs 
determine  if  the  right  things  are  being  done  and  MOPs  determine  if  things  are  being  done 
right.  The  key  difference  between  the  two  is  a  MOP  alone  does  not  provide  indication  of 
normative  behavior.  The  final  measurement,  MOOs  gauge  indirect  conditions  created  by 
the  system. 
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The  key  to  a  successful  measurement  is  ensuring  the  right  measures  are  being 
used  to  gauge  the  system  purpose.  The  challenge  however,  is  differing  between  what  one 
would  like  to  measure  and  what  is  actually  measurable.  Generally  a  vertical  framework  is 
used  for  effectiveness  measures  where  all  measures  are  a  derivative  of  the  systems 
strategic  purpose.  A  real  problem  in  understanding  which  inputs  lead  to  which  outcomes 
is  identifying  and  articulating  the  cause  and  effect  linkage  between  strategic,  operational, 
and  tactical  levels  (Kaplan  &  Norton,  1996).  The  cause  and  effect  relationship  can  be 
difficult  to  discern  because  the  output  of  one  system  could  very  well  be  the  input  of 
another.  Some  systems  can  even  change  overtime  or  adapt  to  being  measured.  The 
primary  goal  is  to  develop  system  measures  that  yield  the  most  insight  while  imposing 
the  least  burden  on  the  system  and  the  person  or  persons  conducting  the  measurement. 

Modeling  large  complex  systems  can  result  in  numerous  measures,  with  each  only 
providing  a  narrow  view  of  the  system.  Having  so  many  different  views  can  make  it 
difficult  to  assess  the  overall  system.  If  this  is  the  case,  aggregation  is  a  tool  that  can  help, 
but  can  be  difficult  because  most  of  the  measurements  are  usually  not  the  same. 
Combining  dissimilar  measurements  requires  an  understanding  of  the  scale  types  being 
used  in  order  to  ensure  the  aggregated  measurement  is  meaningful  and  preserves  the 
original  scale  (Antony,  et  al.,  1998).  One  method  commonly  used  to  combine  measures  is 
the  aggregation  process  which  can  be  additive  or  multiplicative.  The  easiest  and  most 
obvious  is  the  additive: 


Equation  1:  Additive  Aggregation 
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where  is  some  type  of  predetermined  weight  and  is  the  measures.  If  the 
relationship  is  known  to  be  non-linear  you  can  aggregate  using  the  multiplicative 
normalization  process: 


Equation  2:  Multiplicative  Aggregation 

The  last  is  a  higher  order  polynomial  aggregation  which  closely  captures  the  systems 
underlying  nature: 


Equation  3:  Polynomial  Aggregation 

Good  measures  share  six  distinct  characteristics  which  are  timely,  objective, 
economical,  complete,  measurable,  and  strategically  linked  (Bullock,  2006).  A  timely 
measure  is  one  that  is  collected  in  a  time  frame  that  is  relevant.  Objective  measures  are 
measures  that  meet  the  clairvoyance  test,  are  repeatable,  and  have  “face  value”  and  or 
credibility  that  they  actually  represent  the  system.  They  should  also  be  economical  in  the 
sense  that  the  data  or  information  gained  from  creating  the  measure  is  of  more  value  and 
requires  less  effort  than  the  burden  of  the  measurement  activities  themselves.  The 
completeness  characteristic  is  defined  by  a  measure  or  set  of  measures  spanning  the 
entire  system.  A  complete  measure  addresses  both  breadth  and  depth  of  the  attributes  of 
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the  systems  and  the  system  itself.  There  is  no  easy  way  to  achieve  completeness;  this 
requires  creative  and  critical  thinking  and  exceptional  knowledge  of  the  system.  For  a 
measure  to  be  measurable  implies  that  the  measure  can  be  feasibly  obtained  and  the 
collected  measures  are  accurate  and  can  be  verified  (Jordan,  Prevette,  &  Woodward, 
2001). 

For  years  now  senior  executives  in  a  broad  range  of  fields  have  begun  rethinking 
how  to  measure  performance  of  their  businesses.  They  have  all  recognized  that  new 
strategies  and  competiveness  require  new  measurement  systems.  They  have  all  come  to 
the  understanding  that  treating  financial  figures  as  their  only  source  of  performance 
measurement  is  a  flawed  theory  (Eccles,  1991).  Many  mangers  have  been  tracking  things 
such  as  quality,  market  share,  and  other  nonfinancial  measures  for  years,  but  not  using 
them  as  measures  of  performance.  Changing  the  status  quo  has  been  difficult  because 
when  conflicts  arise,  financial  considerations  always  win  out. 

Chief  Executive  Officer’s  now  feel  they  have  initiated  a  change  in  their  business 
practices  in  how  their  managers  think  about  business  performance.  Executives  have  come 
to  the  conclusion  that  what  gets  measured  gets  attention  especially  if  there  is  some  type 
of  reward  tied  to  it.  They  also  understand  that  they  cannot  simply  add  new  measures  to 
the  old  accounting-driven  performance  and  expect  significant  results.  Instead  they  have 
to  identify  key  corporate  performance  measures  such  as  productivity,  employee  attitude 
and  public  responsibility  along  with  managing  short  and  long  term  goals.  Many  in  the 
business  community  blame  the  short-term  thinking  of  most  CEO’s  as  a  major  concern 
when  it  comes  to  change.  The  blame  has  been  cast  on  a  relentless  desire  for  rising 
quarterly  earnings,  while  others  fault  senior  executives  and  their  short  terms  as  the  leader 
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of  the  companies  as  the  reason  for  the  shortsightedness.  This  short-term  thinking  puts 
tons  of  pressure  on  the  managers  themselves  and  they  have  a  strong  incentive  to 
manipulate  the  earnings  reports  (Eccles,  1991).  This  is  a  game  that  few  in  management 
deny  takes  place  and  calls  in  question  the  very  measures  that  the  markets  uses  to 
determine  stock  price. 

Measures  of  Effectiveness  for  governmental  organizations  are  much  more 
difficult  because  their  objective  is  not  necessarily  financial  gain.  The  accounting  systems 
and  economic  and  financial  methods  in  use  in  these  organizations  neither  satisfy  the  large 
information  needs  for  measuring  how  effectively  they  achieve  their  objectives  nor 
provide  the  information  feedback  required  for  high-level  decision  making  about 
allocation  of  budgets  and  resources  (Gawande  &  Wheeler,  1999).  However  the  need  for 
such  measures  of  effectiveness  is  imperative  because  of  the  Chief  Financial  Officers  Act 
of  1990  and  the  Governmental  Performance  Act  of  1990  which  implement  performance 
based  management  across  all  sections  of  the  government.  The  government  is  increasingly 
trying  to  become  more  efficient  and  maximize  its  total  returns  from  its  spending 
allocations. 

2.5  Decision  Analysis 

There  are  multiple  times  in  our  lives  when  we  will  be  faced  with  tough  decisions. 
Most  of  us  make  those  tough  decision  based  on  a  hunch  or  gut  feeling,  but  most  of  us 
wish  we  had  some  way  to  make  those  decisions  in  a  much  easier  systematic  way. 
Decision  analysis  (DA)  provides  a  systematic  structure  and  guidance  for  thinking  about 
hard  decisions  (Clemen  &  Reilly,  2001).  There  are  four  basic  sources  of  difficulty  to  any 
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decision  which  are  complexity,  uncertainty,  multiple  objectives,  and  different 
perspectives.  Complex  problems  are  tough,  but  decision  analysis  provides  effective 
methods  for  organizing  a  complex  problem  into  a  structure  that  can  be  analyzed.  That 
structure  includes  possible  courses  of  action,  possible  outcomes,  the  likelihood  of  those 
outcomes,  and  the  eventual  consequences  (good  or  bad).  In  turn  this  structure  helps 
answer  the  “what  if’  questions  of  complex  problems.  Usually  there  is  no  hard  decision 
made  with  one  hundred  percent  certainty.  DA  helps  identify  important  sources  of 
uncertainty  and  represents  that  uncertainty  in  a  systematic  logical  way  (Clemen  &  Reilly, 
2001).  Many  decisions  have  multiple  objectives  such  as  maximizing  square  footage, 
while  minimizing  cost.  Clearly,  these  objectives  conflict  with  each  other,  but  DA  gives  us 
tools  to  make  trade-offs  when  dealing  with  multiple  objectives.  Finally,  when  there  are 
multiple  decision  makers,  they  rarely  come  to  the  same  conclusion  on  any  decision.  Most 
individuals  will  look  at  a  problem  from  different  perspectives  which  lead  to  different 
choices,  but  DA  once  again  helps  sort  through  and  resolve  these  differences. 

Applying  DA  techniques  correctly  will  help  make  better  decisions,  and  over  time 
produce  better  outcomes.  As  stated  above  there  is  uncertainty  in  any  tough  decision, 
which  means  no  matter  which  decision  that  is  chosen  there  is  some  probability  that  a 
negative  outcome  could  be  the  result.  Additionally,  just  as  there  is  a  chance  of  the 
negative  outcome,  there  is  also  a  possibility  that  you  could  be  lucky  and  have  a  positive 
outcome  when  choosing  a  bad  alternative.  However,  using  DA  consistently  will  improve 
your  chances  of  enjoying  positive  outcomes  and  lessen  the  probability  of  those  negative 
outcomes.  Psychology  has  shown  that  people  generally  do  not  process  information  and 
make  decisions  that  are  consistent  (Clemen  &  Reilly,  2001).  DA  does  not  provide 
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solutions  to  problems,  instead  it  is  an  information  source  that  provides  insight  about  a 
situation,  uncertainty,  objectives,  and  trade-offs,  which  will  yield  some  recommended 
course  of  action.  DA  is  a  tool  in  decision  making  and  is  not  meant  to  replace  the  decision 
maker’s  intuition,  relieve  him  or  her  of  the  obligations  in  facing  the  problem,  or  to  be  a 
competitor  to  the  decision  maker’s  personal  style  of  analysis,  instead  it  is  meant  to 
complement,  augment,  and  generally  work  alongside  the  decision  maker  in  exemplifying 
the  nature  of  the  problem  (Bunn,  1984). 

Many  managers  and  decision  makers  frequently  complain  that  most  analytical 
processes  from  management  science  ignore  subjective  judgment  which  is  the  beauty  of 
DA  because  it  requires  subjective  judgment  (Clemen  &  Reilly,  2001).  Clemen  &  Reilly 
define  their  DA  process  as  seven  step  process  that  begins  with  identifying  the  decision 
situation  and  concludes  with  implementation  of  the  chosen  alternative  (see  figure  3).  As 
stated,  the  first  step  is  for  the  decision  maker  to  identify  the  decision  situation  and  to 
understand  his  or  her  objectives  in  the  situation.  While  there  are  plenty  of  problems  to 
solve,  you  should  avoid  making  a  type  III  statistical  error  in  which  you  do  a  great  job 
solving  the  wrong  problem.  Once  the  problem  has  been  identified,  it’s  time  to  discover 
and  create  alternatives.  The  next  step  in  the  process  modeling  is  the  most  critical  in  DA 
modeling  because  it  enables  users  to  create  quantitative  and  analytical  approaches  to  their 
problems.  This  is  a  key  advantage  to  the  DA  process  because  the  mathematical 
representation  of  the  decision  can  be  subjected  to  analysis.  Choosing  the  best  alternative, 
sensitivity  analysis,  further  analysis  if  needed,  and  implementation  of  the  chosen 
alternative  complete  the  DA  process.  During  these  steps  users  are  attempting  to  answer 
the  “what  if’  questions  about  a  certain  alternative  and  determining  if  slight  changes  in 
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one  or  more  aspects  changes  the  recommended  alternative.  If  small  changes  do  indeed 
change  the  alternative,  the  decision  maker  may  want  to  redefine  certain  objectives, 
include  other  objectives  or  identify  new  alternatives.  As  seen,  the  DA  process  not  only 
provides  a  structured  way  to  think  about  decisions,  but  also  fundamentally  provides 
structure  which  allows  a  decision  maker  to  develop  beliefs,  feelings,  and  those  subjective 
judgments  that  are  critical  for  good  decision  making. 


26 


Figure  3:  Decision  Analysis  7  Step  Process  (Clemen  &  Reilly,  2001) 
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2.  6  Value  Focused  Thinking 


Values  are  what  we  care  about  and  thus  should  be  the  driving  force  for  decision 
making  (Keeney,  1992).  Value  focused  thinking;  a  decision-making  methodology  is  used 
to  ensure  that  decisions  are  made  in  the  most  beneficial  manner  (Pruitt,  2003).  Focusing 
early  and  deeply  on  values  when  facing  difficult  problems  will  lead  to  more  desirable 
consequences,  and  even  to  more  appealing  problems  than  the  ones  we  currently  face 
(Keeney,  1992).  Value-focused  thinking  involves  starting  at  the  best  possible  alterative 
and  working  to  make  it  reality,  while  alternative-focused  thinking  involves  starting  with 
what  is  readily  available  and  taking  the  best  of  the  lot.  Alternative-focused  is  the 
“natural”  way  we  have  learned  to  make  decisions  and  is  deeply  engrained  in  us  to  make 
our  choice  out  of  the  options  available  to  us.  Value-focused  thinking  can  be  thought  of  as 
constraint-free  thinking,  because  we  focus  on  what  we  want  to  achieve  rather  than  the 
selecting  from  alternatives.  Value-focused  thinking  should  lead  both  to  more  appealing 
decision  problems  and  to  choices  among  better  alternatives  than  those  generated  by 
happenstance  or  conventional  approaches  (Keeney,  1992) . 

Any  decision  that  is  a  real  decision,  is  important  to  a  person  or  organization,  and 
is  complex  and  there  is  no  clear  “solution”  is  ideal  for  VFT.  When  faced  with  a  difficult 
decision  start  first  by  thinking  about  your  values  by  writing  down  a  list  of  your 
objectives.  The  principal  of  thinking  about  values  is  to  discover  the  reasoning  of  each 
objective  and  how  it  relates  to  other  objectives  (Keeney,  1992). 
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The  purpose  of  thinking  about  values  is  to  pinpoint  the  values  that  are  the  drivers 
in  a  decision  situation.  Sometimes  you  may  have  a  gut  feeling  about  what  values  are 
relevant,  but  find  them  hard  to  articulate  while  other  times  you  may  have  a  difficult  time 
determining  what  values  are  needed  in  a  complicated  decision.  Figure  4  gives  an 
overview  of  nine  reasons  why  VFT  could  and  would  be  effective  in  any  business, 
government,  or  even  personal  decision  making. 


Before  you  can  get  into  the  steps  or  methodology  of  VFT,  you  must  first  understand  the 
frame  work  of  the  process.  The  decision  context  and  the  fundamental  objectives  together 
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provide  the  decision  frame  (Keeney,  1992).  The  decision  context  defines  the  set  of 
alternatives  appropriate  for  consideration,  while  the  fundamental  objectives  determine  the 
values  in  which  one  cares  about  and  the  class  of  consequence  of  concern.  Better  stated, 
fundamental  objectives  are  the  ends  objectives  of  a  given  decision  context.  Fundamental 
objectives  are  the  basis  of  interest  in  the  decision  being  considered  and  qualitatively  state 
all  that  is  of  concern  in  the  decision  context.  For  example,  the  decision  context  for  a  real 
estate  investor  could  be  what  property  to  purchase.  The  fundamental  objectives  in  this 
context  could  be  price,  square  footage,  neighborhood  and  property  taxes 

Strategic  decision  context  requires  that  you  have  strategic  objectives.  All 
organizations  have  strategic  objectives,  whether  written  down  or  not,  that  help  provide 
common  guidance  to  all  decisions  and  decision  opportunities.  They  also  serve  as  the 
mechanism  by  which  management  can  guide  decisions  by  individuals  or  groups  (Keeney, 
1992).  Structuring  strategic  objectives  can  aid  tremendously  in  decision  making  as  it 
establishes  a  sound  basis  that  can  be  repeatedly  used  and  provides  a  reference  point  for 
even  turbulent  decision  situations. 

As  stated  above,  most  if  not  all  of  us,  are  alternative  focused  thinkers  versus  value 
focused  thinkers.  When  a  decision  opportunity  presents  itself,  the  first  thing  we  do  is 
begin  sorting  through  the  alternatives  we  have  versus  focusing  on  our  values  and 
allowing  those  to  shape  our  alternatives.  There  are  major  short  comings  to  this  method  of 
decision  making  such  as  viable  superior  alternatives  not  being  indentified.  The  objectives 
that  are  identified  are  often  only  means  to  the  consequence  that  are  of  fundamental 
concern  and  there  is  no  logical  match  between  alternatives  and  objectives  (Keeney, 

1992).  Fortunately,  value  focused  thinking  can  significantly  alleviate  these  shortcomings 
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by  allowing  us  to  broaden  the  decision  situation  and  define  it  more  carefully.  This  is  done 
by  not  thinking  about  means  objectives  until  fundamental  objectives  are  found  and  then 
from  the  opposite  direction  work  back  from  strategic  objectives  to  generate  fundamental 
objectives.  This  new  set  of  fundamental  objectives  will  be  much  broader  than  the  means 
objectives,  but  much  narrower  than  the  strategic  objectives,  giving  you  a  well-defined 
decision  frame. 

Solving  decision  problems  is  the  sole  aim  of  alternative-focused  thinking  and  is 
typically  a  reactive  process.  However  you  can  think  of  value-focused  thinking  as  not  only 
a  problem  solving  methodology,  but  as  a  proactive  process  that  helps  with  the 
identification  of  decision  opportunities. 
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Table  1:  Comparing  sequences  of  AFT  &  VFT 


Alternative-Focused  Thinking 

1.  Recognize  a  decision  problem 

2.  Identify  Alternatives 

3.  Specify  values 

4.  Evaluate  alternatives 

5.  Select  an  alternative 


Value-Focused  Thinking 


For  Decision  Opportunities 
Before  specifying  After  specifying 


1.  Recognize  a  decision 
problem 

2.  Specify  values 

3.  Create  alternatives 

4.  Evaluate  alternatives 

5.  Select  alternatives 


strategic  objectives 

1.  Identify  a  decision 
opportunity 

2.  Specify  values 

3.  Create  alternatives 

4.  Evaluate  alternatives 

5.  Select  an  alternative 


strategic  objectives 

1.  Specify  values 

2.  Create  a  decision 
opportunity 

3.  Create  alternatives 

4.  Evaluate  alternatives 

5.  Select  an  alternative 


There  are  five  major  steps  that  are  associated  with  Alternative-Focused  Thinking  that  are 
depicted  in  Table  1.  The  first  three  steps  are  the  big  difference  between  VFT  and  AFT. 
Step  one  of  AFT  “Recognize  a  decision  problem”  usually  takes  place  as  a  result  of 
actions  out  of  the  control  of  the  decision  maker  and  is  generally  a  plea  for  something  to 
be  done.  Step  two  is  to  “identify  the  alternatives.”  Sometimes  this  can  be  as  easy  as 
turning  the  light  on  or  leaving  it  off.  Regardless  of  the  decision  context  all  the  alternatives 
are  almost  always  already  known  prior  to  making  the  decision.  In  some  instances 
decision  makers  attempt  to  search  for  additional  alternatives,  but  the  stated  alternatives 
anchor  the  thought  process  and  stifle  creativity  and  innovation.  The  third  step  of  AFT  is 
typically  done  with  much  less  thought  than  one  would  expect  with  the  VFT  process. 

Since  alternatives  are  already  identified,  values  are  selected  based  on  the  alternatives 
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available  and  no  real  thought  about  the  fundamental  objectives  take  place  during  this 
stage. 

2.7  Summary 

Intelligence  is  a  stochastic  process  and  it  is  difficult  to  know  with  any  sort  of 
certainty  when  a  valuable  piece  of  information  will  present  itself.  Commanders  are  and 
have  been  aware  of  this  fact  for  years  but  still  thrust  their  assets  into  situations  that  are 
less  than  optimal  to  try  gain  an  edge  in  intelligence.  GMTI  onboard  JSTARS  is  one  of 
those  capabilities  that  has  been  used  in  optimal  and  less  than  optimal  conditions. 

Through  its’  successes  and  failures  intelligence  analyst  have  gained  valuable 
knowledge  on  how  to  successfully  employ  JSTARS.  Using  this  knowledge  and  the 
knowledge  of  how  to  create  MOEs  that  can  effectively  measure  a  system,  this  research 
will  help  decision  makers  use  their  dwindling  assets  more  effectively  and  increase  the 
value  of  information  they  receive. 

AFT  is  the  decision  making  process  that  most  people  undertake  when  a  decision 
problem  is  presented.  The  previous  material  has  shown  how  there  are  numerous  short 
comings  with  making  decisions  in  this  manner.  This  research  will  help  move  decision 
makers  from  AFT  to  VFT  in  order  to  help  them  make  decisions  that  are  quantifiable, 
repeatable,  and  take  into  consideration  the  values  of  the  objectives  they  are  trying  to 
achieve.  Chapter  3  will  further  define  and  develop  the  VFT  process  and  demonstrate 
how  this  methodology  can  help  all  involved  make  better  decisions  when  it  comes  to  using 
intelligence  assets. 
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Chapter  3.  Methodology 


As  stated,  JSTARS  has  had  many  success  and  many  failures  over  the  years.  In 
each  situation  there  were  key  factors  that  enabled  the  system  to  fail  or  succeed.  Using  the 
VFT  methodology  we  will  be  able  to  generate  scenarios  that  will  almost  always  produce 
positive  results.  If  a  decision  maker  decides  to  fly  missions  that  don’t  perform  well  in  the 
model,  they  will  know  before  the  mission  is  ever  flown  that  the  probability  of  getting 
high  values  information  on  said  mission  will  be  exceptionally  low.  In  this  chapter  the 
VFT  process  will  be  described  in  greater  detail.  Specifically,  the  10  steps  of  the  VFT 
process  will  be  expounded  upon.  Terms  that  will  be  important  to  know  in  this  chapter  and 
referenced  often  are  listed  below  in  Table  2. 


Table  2:  VFT  Key  Terms 


Evaluation  Consideration 

Any  matter  that  is  significant  enough  to  be  taken  into 
account  while  evaluating  alternatives. 

Objective 

The  preferred  direction  of  movement  with  respect  to  an 
evaluation  consideration.  Assumes  that  preference  displays  a 
monotonic  behavior  which  means  either  “more  is  better”  or 
“less  is  better”  with  respect  to  each  evaluation  consideration. 

Goal 

The  threshold  of  achievement  with  respect  to  an  evaluation 
consideration  which  is  either  attained  or  not  by  any 
alternative  that  is  being  evaluated. 

Evaluation  Measure 

A  measuring  scale  for  the  degree  of  attainment  of  an 
objective.  Example  “annual  salary  in  dollars” 

Level  or  Score 

A  numerical  rating  for  a  particular  alternative. 

Value  Structure 

The  entire  set  of  evaluation  considerations,  objectives,  and 
evaluation  measures  for  a  particular  decision  analysis. 

Value  Hierarchy  or  Tree 

A  value  structure  with  hierarchal  a  “treelike”  structure. 

Layer  or  Tier 

The  evaluation  consideration  at  the  same  distance  from  the 
top  of  a  value  hierarchy. 

3.1  Introduction 
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The  current  value  model  process  was  created  and  compiled  by  Shovaik  (Shoviak, 
2001)  and  is  broken  down  into  10  steps  which  is  depicted  in  Table  3.  The  first  and 
probably  most  crucial  part  of  the  VFT  process  is  identifying  the  problem.  Once  the 
correct  problem  has  been  identified,  it  is  now  time  to  create  the  value  hierarchy.  This  step 
entails  sitting  down  with  the  decision  maker  or  decision  makers  and  finding  out  what  are 
the  things  that  they  value  or  what  is  important  about  the  particular  decisions.  For 
example,  if  you  were  purchasing  a  new  home  one  of  the  things  that  would  be  of  value  to 
most  people  would  be  price.  Section  3.3  will  discuss  the  procedures  for  developing  a 
value  hierarchy. 

Once  the  decision  maker  is  satisfied  they  have  captured  everything  that  is 
important  with  the  objective  of  the  decision  it’s  time  to  move  on  to  step  3  of  the  process 
which  is  creating  evaluation  measures.  Using  the  house  example  again,  assume  location 
was  in  the  value  hierarchy,  what  things  about  the  location  are  important.  Is  it  being  close 
to  your  child’s,  school,  being  close  to  work  or  shopping  malls,  having  sidewalks,  high 
property  values,  or  is  it  all  of  the  above.  Section  3.4  will  go  into  greater  detail  on  how  to 
determine  effective  measures.  The  creation  of  value  functions  is  the  next  step  in  the 
process.  During  this  step  a  single  dimensional  value  functions  will  be  assigned  to  each 
measure  which  will  assign  a  score  to  each  alternative  and  will  be  discussed  in-depth  in 
section  3.5. 

The  weighting  of  the  hierarchy  is  the  next  step  in  the  process.  In  this  step  the 
decision  maker  will  have  to  determine  how  much  weight  to  give  to  each  value  in  the 
hierarchy.  This  is  an  important  step  because  it  is  when  the  DM  determines  which 
measures  are  most  important  and  which  are  least  important.  It  is  important  to  note  that 
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steps  1  -5  all  require  input  from  your  decision  maker  and/or  subject  matter  experts.  The 
remaining  steps  can  and  should  be  completed  by  the  analyst  without  any  input  from  the 
decision  maker. 

Generating  alternatives  is  the  next  step  in  the  process  and  is  completed  by 
populating  the  model  with  a  fully  exhaustive  list  of  alternatives.  Once  the  alternatives 
have  been  generated  it’s  time  to  score  each  alternative.  This  process  is  done  by  scoring 
each  alternative  against  every  measure  in  the  hierarchy.  Once  the  scoring  has  been 
completed,  the  deterministic  analysis  takes  place  by  multiplying  the  score  in  the 
particular  measure  against  the  weight  that  was  given  by  the  DM  to  come  up  with  an 
overall  raw  score  for  each  alternative.  Sensitivity  analysis  is  then  done  on  the  model  to 
determine  if  small  changes  in  the  weight  values  will  cause  the  ranking  of  the  alternatives 
to  change.  The  final  step  is  to  communicate  the  conclusions  of  the  analysis  and 
recommend  a  course  of  action  to  the  decision  maker. 


36 


Table  3:  10  Step  VFT  Process  (Shoviak,  2001) 


Step  1. 

Problem  Identification 

Step  2. 

Create  the  Value  Hierarchy 

Step  3. 

Develop  the  Evaluation  Measures 

Step  4. 

Create  the  Value  Functions 

Step  5. 

Weight  the  Hierarchy 

Step  6. 

Generate  Alternatives 

Step  7. 

Alternative  Scoring 

Step  8. 

Deterministic  Analysis 

Step  9. 

Sensitivity  Analysis 

Step  10. 

Conclusions  and  Recommendations 

3.2  Step  1:  Problem  Identification 

The  problem  identification  step  is  one  of  the  most  important  steps  in  this  entire 
process.  It  would  be  a  shame  to  go  through  this  entire  process  to  learn  at  the  out  brief  to 
your  decision  maker  that  you  have  committed  a  type  III  error  and  solved  the  wrong 
problem.  Sometimes  problem  identification  can  be  quite  evident  when  deciding  which 
car  to  purchase  or  which  house  to  buy.  At  other  times,  it  may  take  a  little  time  to  get 
down  to  the  root  cause  of  the  problem.  This  is  why  it  is  imperative  to  take  the  time  up 
front  to  determine  what  the  true  problem  is,  because  if  not,  the  resulting  solution  will 
have  no  value  and  be  considered  a  wasted  effort  (Jurk,  2002). 
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Buy  a  Dishwasher 
(1.00) 


Figure  4:  Dishwasher  Example  Hierarchy 
3.3  Step  2:  Creation  of  the  Value  Hierarchy 

The  value  hierarchy  serves  as  the  apparatus  that  allows  the  decision  maker  to 
evaluate  each  alternative.  The  model  structures  the  values  that  the  decision  maker  has 
concluded  to  be  important  in  context  to  their  decision  and  uses  some  type  of  measure 
process  to  evaluate  how  each  alternative  scores.  The  hierarchy  gives  decision  makers  a 
repeatable  and  defendable  decision  making  support  and  enables  them  to  identify  possible 
missing  values.  The  hierarchy  should  without  a  doubt  be  collectively  exhaustive  and 
mutually  exclusive,  or  in  other  words  every  value  that  is  important  should  be  explored 
and  no  two  values  or  measures  should  represent  the  same  thing. 
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3.3.2  Properties  of  the  Hierarchy 


The  desirable  properties  of  a  hierarchy  are  completeness,  non-redundancy, 
decomposability,  operability,  and  small  size  (Kirkwood,  1997).  A  complete  hierarchy  is 
one  that  adequately  covers  all  concerns  necessary  to  evaluate  the  overall  objective  of  the 
decision.  Completeness  ensures  that  the  alternatives  are  adequately  evaluated  and  ranked 
accordingly. 

A  non-redundant  hierarchy  is  one  where  no  two  evaluation  considerations  in  the 
same  layer  or  tier  of  the  hierarchy  overlap.  For  example,  in  Figure  4  the  cost  of  the 
dishwasher  is  divided  into  purchase  price  and  installation  cost.  For  this  hierarchy  to  be 
non-redundant  every  cost  associated  with  the  dishwasher  should  fit  one  of  these  two 
categories. 

Decomposability  which  is  better  known  as  Independence  means  that  the  score  an 
alternative  receives  should  not  influence  its  score  in  another  measure.  This  property  is 
easier  explained  with  an  example  illustrated  by  Kirkwood.  Assume  a  “value  of 
economics”  issue  with  lower  tier  values  of  “salary”,  “pension  benefits”  and  “medical 
coverage.”  Note  that  for  the  lower  tier  values,  the  “value  attached  to  the  variations  in 
scores  depends  on  the  levels  of  the  other  two  lower  tier  values.”  Simple  stated,  if  the 
salary  were  $250,000  a  year,  there  would  be  no  value  to  a  slight  increase  in  “pension 
benefits”  and  “medical  coverage.”  Therefore,  the  values  are  not  independent  (Kirkwood, 
1997). 

Operability  means  that  the  hierarchy  is  understandable  for  the  person  or  persons  that 
are  using  it.  Operability  generally  becomes  a  problem  when  technical  specialists  have  to 
interact  with  the  general  public.  A  great  example  of  this  is  when  technical  experts  had  to 
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interact  with  the  public  during  the  Three  Mile  Island  nuclear  power  plant  incident.  During 
the  analysis  of  the  event,  experts  had  a  very  difficult  time  presenting  an  assessment  of 
risk  to  journalist  and  the  general  public.  In  general,  it  is  better  to  compromise  on  some 
aspects  of  the  hierarchy  in  order  to  create  evaluation  measures  that  are  operable  and  easy 
to  understand. 

The  final  desirable  property  of  a  hierarchy  is  that  it  be  of  small  size.  A  smaller  value 
hierarchy  can  be  communicated  more  easily  to  interested  parties  and  requires  fewer 
resources  to  estimate  the  performance  of  alternatives  with  respect  to  the  various 
evaluation  measures  (Kirkwood,  1997).  Many  business,  government,  and  not-for-profit 
groups  have  a  tendency  to  keep  adding  evaluation  considerations  until  the  hierarchy 
becomes  so  complex  that  it  becomes  difficult  for  an  analyst  to  conduct  and  interpret.  The 
quest  for  completeness  and  detail  must  be  balanced  against  the  need  to  finish  the  analysis 
in  a  manageable  time  frame  and  budget.  When  faced  with  this  issue  analyst  should  use 
the  “test  of  importance.”  This  test  states  that  an  evaluation  measure  should  be  included 
only  if  possible  variations  among  the  alternatives  with  respect  to  the  proposed  evaluation 
could  change  the  preferred  alternative  (Kirkwood,  1997).  For  example,  if  you  were 
purchasing  a  hat  and  all  colors  but  red  were  acceptable,  it  probably  would  not  be  prudent 
to  add  color  to  the  hierarchy  since  hats  come  in  multiple  colors. 

3.3.2  Hierarchy  Structure 

There  are  a  couple  different  approaches  to  developing  or  structuring  a  hierarchy.  The 
method  for  developing  a  hierarchy  is  dependent  on  whether  the  alternatives  are  known  at 
the  time  the  hierarchy  is  being  developed.  If  the  alternatives  are  known,  then  a  bottom  up 
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approach  is  appropriate,  if  not  the  top-down  strategy  is  more  appropriate.  Most  of  the 
time  it  is  necessary  to  build  a  specific  hierarchy  to  solve  your  problem  because  creating  a 
general-purpose  hierarchy  which  would  solve  a  wide  range  of  problems  is  complex  and 
impractical.  However,  since  value  modeling  has  been  around  for  several  decades,  you  can 
sometimes  find  and  use  a  previously  used  hierarchy  that  fits  your  problem  instead  of 
starting  from  scratch. 

In  “bottom-up”  or  “alternative  driven”  alternatives  are  examined  to  determine  the 
ways  in  which  they  differ.  The  evaluation  measures  are  then  developed  to  evaluate  things 
in  which  the  alternatives  differ.  This  approach  develops  the  bottom  layer  of  the  hierarchy, 
and  then  constructs  the  remainder  of  the  hierarchy  on  top  of  this  layer. 

The  “top-down”  or  “objective-driven”  is  used  when  alternatives  are  not  as  well 
known.  The  process  starts  with  an  overall  objective  and  subdivides  as  appropriate  to 
develop  the  bottom  tiers.  One  of  the  main  purposes  of  this  method  is  to  identify  potential 
alternatives.  Also  by  starting  with  an  overall  objective  and  subdividing  it  helps  develop 
the  evaluation  considerations  in  greater  detail.  This  is  also  the  preferred  method  of  most 
VFT  modelers. 

3.3.3  Standards  of  Information 

In  soliciting  information  about  the  hierarchy  from  decision  makers  and  stake  holders 
there  are  three  standards,  Gold,  Silver  and  Platinum  (Weir,  2010).  The  gold  standard  is 
the  lowest  of  the  three  and  entails  using  the  decision  maker’s  strategic  vision  or  plan  to 
deductively  develop  the  value  hierarchy.  The  next  standard,  the  silver  standard,  entails 
having  meetings  with  a  large  group  of  stakeholders  to  inductively  develop  the  value 
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hierarchy  using  affinity  diagrams.  The  final  and  best  way  to  solicit  information  is  the 
Platinum  standard.  This  includes  interviewing  senior  leaders  and  key  technical  personnel 
to  again  inductively  develop  the  value  hierarchy  via  affinity  diagrams  (Weir,  2010).  This 
is  the  best  way  to  get  the  information,  but  also  the  most  difficult  since  senior  leaders  do 
not  usually  have  time  to  sit  down  with  an  analysis  and  describe  exactly  what  he  or  she 
wants  multiple  times. 

3.3.4  Affinity  Diagrams 

An  affinity  diagram  is  a  tool  that  gathers  large  amounts  of  data  (ideas,  opinions, 
issues)  and  organizes  them  into  groups  based  on  the  nature  of  their  relationships 
(Defense,  Basic  Tools  for  Process  Improvement:  Module  4  Affinity  Diagram,  2007).  The 
affinity  process  is  a  proven  way  to  get  people  to  work  on  creative  level  to  address 
difficult  issues.  The  process  is  extremely  useful  when  sifting  though  large  volumes  of 
data  because  it  allows  team  members  to  organize  the  data  into  groups.  It  is  also  useful 
when  attempting  to  encourage  new  patterns  of  thinking.  Since  brainstorming  is  the  first 
step  in  the  process  the  team  considers  all  ideas  from  all  members  without  criticism.  This 
often  stimulates  a  creative  list  of  ideas  and  allows  members  to  break  away  from  the 
traditional  entrenched  thinking. 

When  creating  affinity  diagrams  there  are  three  basic  tenets  that  discussion  leaders 
should  always  abide  by.  The  first  is  “Do  it  silently.”  The  most  effective  way  to  work  is  to 
have  everyone  move  items  at  will,  without  talking.  This  helps  encourage  unconventional 
thinking,  discourages  semantic  battles  and  prevents  one  person  from  steering  the  affinity. 
The  second  tenet  is  “Go  for  the  Gut  Reactions.”  This  tenet  encourages  team  members  to 
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react  quickly  as  speed  rather  than  deliberation  is  most  important  to  keep  the  process 
moving.  The  final  tenet  is  “Handle  Disagreements  Simply.”  When  a  team  member  does 
not  agree  where  an  idea  is  grouped  allow  them  to  move  it.  If  consensus  still  cannot  be 
reached,  create  a  duplicate  and  place  one  in  each  group.  This  creates  an  environment 
where  it  is  okay  to  disagree. 


Table  4:  Steps  to  Creating  an  Affinity  Diagram  (Defense,  Basic  Tools  for  Process  Improvement: 

Module  4  Affinity  Diagram,  2007) 


Creating  an  Affinitv  Diagram 

Step  1 

Generate  Ideas 

Step  2 

Display  Ideas 

Step  3 

Sort  Ideas  Into  Groups 

Step  4 

Create  Header  Cards 

Step  5 

Draw  Finished  Diagram 

Creating  affinity  diagrams  involves  a  five  step  process  (See  Table  4).  The  first  step 
“Generate  Ideas”  is  the  brain  storming  session  where  all  ideas  are  written  on  post-its. 

Step  2  “Display  the  Ideas”  simply  post  all  the  ideas  generated  in  a  random  order  on  a 
board  or  table.  “Sorting  the  Ideas  into  Groups”  is  when  team  members  do  so  without 
talking.  They  do  this  by  looking  for  two  ideas  that  seem  related  and  placing  them 
together.  This  process  is  repeated  until  all  ideas  have  been  placed  in  a  group.  (If  there  are 
ideas  that  don’t  fit  into  any  group,  let  them  stand  alone  under  their  own  headers  (Defense, 
Basic  Tools  for  Process  Improvement:  Module  4  Affinity  Diagram,  2007).)  Next  is  to 
“Create  header  cards  for  the  groups.”  A  header  is  an  idea  that  captures  the  essential  link 
among  the  ideas  contained  in  the  group.  The  final  step  in  the  process  is  to  “Draw  the 
finished  Affinity  Diagram.”  Write  down  the  problem  statement,  place  headers  and  super 
header  cards  above  the  groups,  review  and  clarify  groupings  and  document  the  finished 
affinity  diagram. 
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3.4  Step  3:  Develop  Evaluation  Measures 


Evaluation  measures,  also  called  “measures  of  effectiveness,”  “attributes”  or 
“metrics”  allow  an  unambiguous  rating  of  how  well  an  alternative  does  with  respect  to 
each  objective. 


3.4.1  Types  of  Evaluation  Measure  Scales 


Table  5:  Types  of  Evaluation  Measure  Scales 


Natural 

Constructed 

Direct 

Commonly  understood  measures  directly  linked  to 
strategic  objectives  -  Example:  Profit 

Measures  directly  linked  to  the  strategic  objective 
but  developed  for  a  specific  purpose  -  Example: 
Figure  Skating 

Proxy 

In  general  use  the  measure  focused  on  an 
objective  correlated  with  the  strategic  objective  - 
Example:  Gross  National  Product 

Measures  developed  for  a  specific  purpose 
focused  on  an  objective  correlated  to  the  strategic 
objective  -  Example:  Student  Grades 

Evaluation  measures  can  be  classified  as  either  natural  or  constructed  and  direct 
or  proxy  (see  table  5).  A  natural  scale  is  in  general  use  with  a  common  interpretation  by 
everyone.  A  good  example  would  be  “number  of  fatalities”  which  is  a  natural  scale  for 
evaluating  death.  A  constructed  scale  is  one  that  is  developed  for  a  particular  decision 
problem  to  measure  the  degree  of  attainment  of  an  objective.  These  are  typically  used 
when  natural  scales  are  not  appropriate.  A  direct  scale  is  one  that  directly  measures  the 
degree  of  attainment  of  an  objective,  while  a  proxy  scale  reflects  the  degree  of  attainment 
of  its  associated  objective,  but  does  not  directly  measure  this  (Kirkwood,  1997).  There 
are  many  questions  that  arise  when  developing  evaluation  measures  such  as  should  the 
scale  be  a  natural  proxy  or  a  constructed  direct?  Should  the  scales  be  subdivided  to 
provide  further  detail,  or  how  carefully  should  you  specify  the  scale  definition  of  a 
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constructed  scale?  Whatever  scale  you  choose,  the  goal  is  to  make  sure  that  it  is  not 
ambiguous.  The  best  scales  always  pass  the  clairvoyance  test  in  that  if  there  were  a 
clairvoyant  that  could  foresee  the  future  with  no  uncertainty;  they  would  be  able  to 
unambiguously  assign  a  score  to  the  outcome  from  each  alternative.  Most  natural 
measures  easily  pass  the  clairvoyance  test,  but  constructed  scales  can  be  more  difficult  to 
develop  to  do  this. 

3.5  Step  4:  Creating  Value  Functions 

Each  measure  that  was  created  in  the  previous  step  has  to  have  some  mechanism 
to  properly  analyze  each  alternative  and  give  it  a  score.  The  mechanism  that  is  used  to  do 
this  is  the  Single  Dimensional  Value  Function.  The  SDVF  enables  a  combination  of 
multiple  evaluation  measures  into  a  single  index  of  the  overall  desirability  of  an 
alternative  (Kirkwood,  1997).  This  is  done  by  having  the  SDVF  vary  between  zero  and 
one  over  the  range  of  the  scores  of  interest.  This  allows  an  alternative  with  the  most 
preferred  option  to  have  a  score  of  one  and  the  alternative  with  the  least  preferred  option 
to  have  a  score  of  zero. 

3.5.1Types  of  Singe  Dimensional  Value  Functions 

There  are  two  different  types  of  SDVF’s  that  will  be  discussed  in  the  section.  The 
first  is  the  piecewise  linear  function  which  is  made  up  of  segments  of  straight  lines  that 
are  joined  together.  The  second  is  the  exponential  that  uses  a  specific  mathematical  form. 

The  piecewise  linear  function  is  most  practical  when  the  evaluation  measure 
being  considered  has  a  small  number  of  possible  scoring  levels.  In  order  to  determine  the 
piecewise  linear  function  it  requires  that  the  relative  value  increments  be  specified 
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between  each  of  the  possible  evaluation  measure  scores  (Kirkwood).  Since  all  values  are 


between  0  and  1,  Figure  5  shows  an  example  of  a  piecewise  hnear  function.  In  the 
example,  notice  that  if  the  alternative  x-axis  score  falls  under  “choice  1”  it  receives  no 
points  and  for  that  same  measure  if  the  x-axis  score  falls  under  “choice  5”  the  alternative 
receives  all  the  points  for  that  particular  measure. 


1 1 1 

Choice  1  Choice  2  Choice  3  Choice  4  Choice  5 

Figure  5:  Monotonically  Increasing  Piecewise  Linear  Function 

Sometimes  it  is  extremely  impractical  to  use  a  piecewise  linear  SDVF  because  of 
the  large  number  of  value  increments  that  would  have  to  be  found.  In  these  cases,  it’s 
more  appropriate  to  use  an  exponential  SDVF  .The  exponential  SDVF  is  used  when  the 
evaluation  measure  being  considered  can  take  on  an  infinite  number  of  possible  scoring 
levels  as  depicted  in  Figure  6.  The  exponential  function  has  a  particular  form  that 
depends  on  the  range  of  the  evaluation  measure  and  an  exponential  constant  denoted  by 
the  Greek  letter  p  (rho).  The  shape  of  the  exponential  SDVF  is  dependent  upon  the  value 
of  p. 
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Figure  6:  Monotonically  Increasing  Exponential  SDVF 

As  p  increases  the  shape  of  the  graph  becomes  less  curved  until  it  becomes  a  straight  line 
with  infinitely  large  values.  If  the  preferences  are  monotonically  increasing  over  an 
evaluation  measure  x  (that  is,  higher  amounts  of  x  are  preferred  to  lower  amounts)  then 
use  the  equation  in  (Equation  4). 


Equation  4:  Monotonically  Increasing  Equation 

If  preferences  are  monotonically  decreasing  over  x  (that  is,  lower  amounts  of  x  are 
preferred  to  higher  amounts)  then  use  the  equation  in  (Equation  5)  where  “Low”  is  the 
lowest  level  of  x  of  interest ,  “High”  is  the  highest  level  and  p  is  the  exponential  constant 
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(Kirkwood,  1997).  In  a  monotonically  increasing  function  the  v(Low)  =  0  and  the  v(High) 
=  1.  In  a  monotonically  decreasing  function  the  v(Low)  =  1  and  the  v(High)  =  0. 


Equation  5:  Monotonically  Decreasing  Equation 

The  appropriate  value  of  p  depends  on  the  range  of  the  possible  scores  for  the  evaluation 
measure.  In  particular,  realistic  values  of  p  will  generally  have  a  magnitude  greater  than 
one-tenth  of  the  range  of  the  possible  scores  (Kirkwood,  1997).  For  instance,  if  the 
possible  values  range  from  0  to  10  a  realistic  value  of  p  would  be  1  or  greater  if  positive 
and  -1  or  less  if  negative.  There  is  no  upper  limit  for  the  magnitude,  but  once  again  as  p 
grows  infinitely  large  the  value  function  curve  will  be  straight. 

3.6  Step  5:  Weighting  the  Value  Hierarchy 

The  final  step  of  the  value  model  that  requires  DM  or  stakeholder  input  are  the 
weights.  The  weights  are  especially  important  in  determining  which  alternatives  will 
score  the  best.  It  is  crucial  to  work  closely  with  the  DM  to  get  the  best  set  of  weights 
possible.  If  the  DM  is  unsure  about  some  of  their  weights  they  will  have  an  opportunity 
during  sensitivity  analysis  to  find  out  how  sensitivity  their  choices  are  and  what  changes 
can  lead  to  different  decisions.  During  this  step  the  DM  determines  the  relative 
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importance  of  each  value  and  measure  in  their  hierarchy.  When  weighting  the  hierarchy 
there  are  a  few  terms  that  one  should  be  familiar  before  beginning.  Those  terms  are 
branches,  tiers,  local  weights  and  global  weights.  Below  in  Figure  7  the  oval  labeled  as 
“Branch”  depicts  a  branch  of  the  hierarchy.  Each  value  in  a  hierarchy  should  have  a 
branch  associated  with  it  that  goes  down  to  the  lowest  tier  of  the  branch  which  should  be 
the  evaluation  measures.  The  next  word  is  tier. 


Figure  7:  Tiers  &  Branches  of  a  Hierarchy  (Weir,  2010) 


Tier  3 


The  evaluation  considerations  at  the  same  distance  from  the  top  of  a  value  hierarchy 
constitute  a  “layer”  or  “tier”  (Kirkwood,  1997).  Global  weights  sum  to  1  across  an  entire 
tier  and  are  calculated  from  the  local  weights  (Weir,  2010).  In  Figure  8  below  notice  that 
the  numbers  across  the  bottom  sum  to  1.  They  are  calculated  by  multiplying  the  local 
weight  in  the  2nd  tier  above  times  the  local  weight  in  the  tier  3ld.  For  example,  in  the  first 
branch  multiple  .30  *  .20  and  you  will  get  .06,  its  global  weight.  Global  weights  are  used 
when  using  a  bottom-up  approach. 
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Figure  8:  Global  &  Local  Weights  (Weir,  2010) 


Local  weights  sum  to  1  on  a  tier  within  a  branch  and  are  calculated  from  the  global 
weights  (Weir,  2010).  For  instances,  in  the  first  branch  of  the  hierarchy  in  figure  8  .2  +  .8 
sum  tol  and  are  the  local  weights  of  this  particular  branch. 

3.6.1  Techniques  to  Determine  the  Weights 

There  are  several  techniques  used  to  solicit  the  weights  for  the  hierarchy.  One 
way  is  the  “group  weight  assessment  procedure”  or  “direct  assessment.”  In  many 
situations  the  weights  are  accessed  using  a  group  of  people.  In  this  process  each  person 
spreads  100  points  (can  be  poker  chips,  pennies,  etc)  which  equates  to  100%  of  the 
weight  among  the  different  evaluation  considerations.  Once  everyone  has  allocated  their 
weight  to  the  hierarchy,  the  discussion  leader  calculates  the  average  weights.  After 
calculation,  discussion  takes  place  of  any  significant  differences.  Once  discussion  is 
complete,  a  revote  is  taken  and  if  there  are  no  major  differences  then  these  are  the 
weights  for  the  hierarchy. 


50 


A  second  method  is  to  build  a  Swing  Weight  Matrix.  In  this  method  a  swing 
weight  matrix  like  in  Figure  9  is  built.  Next,  with  DM  or  stakeholder  input,  the  values  of 
each  row  are  filled  in  with  a  number  which  indicates  its  importance.  Next,  each  measure 
is  placed  in  its  appropriate  box.  After  all  measures  are  in  their  correct  position  the 
weights  are  calculated  as  a  ratio  of  box,y/sum  of  all  boxes  used. 
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Figure  9:  Swing  Weight  Matrix 


A  final  way  to  calculates  weights  is  via  the  Analytic  Hierarchy  Process  or  AHP. 

In  order  successfully  complete  this  process  a  pair  wise  comparison  of  the  measures  to  be 
weighted  must  be  built.  The  next  step  is  to  judge  the  relative  importance  of  each  measure 
within  a  pair.  Then,  a  comparison  matrix  is  built  and  the  max  Eigen-value  and 
Eigenvector  is  calculated.  Once  the  Eigenvector  is  normalized  you  have  the  weights.  This 
process  seems  more  difficult  than  it  really  is.  Today  there  are  software  packages  that  can 
help  do  this  process.  An  example  of  what  one  of  these  software  packages  would  look  like 
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is  in  figure  10.  Here  it  shows  that  two  measures  can  be  compared  to  each  other  one  at  a 
time.  Figure  10  reads  as  measure  1  is  more  important  than  measure  2,  measure  1  is  more 
important  than  measure  3,  and  measure  3  is  more  important  than  measure  2. 
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Figure  10:  AHP  Example 
3.7  Step  6:  Alternative  Generation 

Keeney  states  “The  range  of  alternatives  people  identify  for  a  given  decision 
situation  is  often  unnecessarily  narrow  (Keeney,  1992).”  This  is  mostly  caused  by  a  need 
to  feel  progress  toward  reaching  a  solution  to  the  decision  problem.  The  genius  of  VFT  is 
that  it  is  considered  to  be  constraint-free  thinking.  This  method  allows  freedom  to 
consider  options  that  normally  would  not,  and  then  allow  the  model  to  determine  which 
one  objectively  does  the  best  in  meeting  the  objectives. 

Often  in  decision  making  opportunities  there  is  always  the  “do  nothing”  option  or 
“status  quo.”  Regardless  if  this  is  the  best  option  or  not,  most  of  the  time  this  is  the 
anchor  point  for  creating  more  alternatives  which  limits  the  search  to  similar  alternatives. 
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This  tendency  can  be  counteracted  by  beginning  the  search  for  alternatives  at  the  “ideal 
point”  or  best  hypothetical  alternative  and  then  down  grading  that  alternative  until  it 
reaches  the  feasibility  region.  There  are  usually  several  different  anchors  in  terms  of 
consequences  that  can  be  used  in  a  given  problem  (Keeney,  1992).  Each  anchor  should 
search  different  places  in  the  mind  for  alternatives. 

Kirkwood  suggests  that  thought  is  an  “associative  process”  and  people  think 
about  a  new  situation  by  making  mental  associations  with  previous  situations  that  seem 
relevant.  These  associations  occur  with  relative  little  conscious  control  an  ideas  “pop  into 
our  minds”  and  they  are  used  as  a  basis  for  structuring  our  consideration  of  the  new 
situation  (Kirkwood,  1997). 

3.7.1  Method  for  Generating  Alternatives 

There  are  a  few  different  ways  to  develop  good  alternatives.  One  way  is  to 
develop  them  based  off  the  lowest  tier  of  the  hierarchy  one  at  a  time  (Kirkwood,  1997). 
This  is  done  by  developing  alternatives  that  do  well  in  one  of  the  evaluation  criteria  while 
not  considering  the  others.  The  alternatives  generated  are  typically  too  one-dimensional 
to  be  feasible,  but  they  allow  a  combination  of  the  strong  points  of  each  to  make  better 
alternatives. 

A  hybrid  approach  to  the  first  option  is  to  consider  multiple  objectives.  This 
approach  is  started  by  considering  two  objectives  at  a  time.  The  alternatives  created  now 
are  likely  to  be  refinements  or  combinations  of  those  created  using  single  objectives 
(Keeney,  1992).  Then  take  three  objectives  at  a  time,  then  four  and  so  on,  until  all 
objectives  have  been  considered  together.  The  final  step  is  to  examine  the  alternatives 
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that  have  been  generated  to  see  if  it  is  possible  to  combine  any  of  them  into  a  single 
alternative.  Again,  these  alternatives  may  not  be  feasible  either. 

Another  method  is  to  maximize  objectives  at  a  higher  tier  in  the  hierarchy 
(Kirkwood,  1997).  This  method  is  likely  to  generate  alternatives  that  are  more  balanced 
than  ones  generated  by  focusing  on  the  lower  tiered  objectives. 

3.7.2  Number  of  Alternatives 

In  some  cases  there  are  far  too  many  alternatives  and  in  others  there  may  be  far 
too  few.  In  this  section  we  will  briefly  discuss  some  methods  to  increase  or  decrease  the 
number  of  alternatives  generated. 

Having  a  large  number  of  alternatives  generally  presents  two  problems.  Primarily, 
it  is  difficult  to  organize/evaluate  information  about  the  alternatives  and  secondly  it  is 
extremely  difficult  in  some  situations  to  collect  the  required  information  about  the 
potential  alternatives  (Kirkwood,  1997).  In  some  situations  there  are  literally  an  infinite 
number  of  alternatives.  For  instances,  if  there  was  a  value  hierarchy  that  composed  of 
several  exponential  single  dimensional  value  functions  it  would  be  virtually  impossible  to 
enumerate  every  possible  combination  as  each  exponential  SDVF  has  an  infinite  amount 
of  choices.  In  many  portfolio  problems  combinational  growth  can  grow  rapidly.  The 
number  of  -combinations  of  a  set  with  elements  is  represented  by 


Equation  6 
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Having  only  10  different  alternatives  will  generate  1023  possible  combinations.  A 
method  to  reduce  the  number  of  alternatives  is  to  use  screening  criteria.  Screening  the  list 
of  alternatives  to  marginally  reduce  the  size  of  the  alternative  pool  can  greatly  reduce  the 
number  of  combinations  and  thereby  the  time  and  costs  associated  with  evaluation  (Cote, 
2010).  Using  the  dishwasher  hierarchy  in  Figure  5,  a  good  example  of  screening  criteria 
will  be  illustrated.  Say  for  instances  you  only  had  $500  to  purchase  the  new  dishwasher. 

It  would  be  feasible  to  screen  out  dishwashers  over  $550  as  you  probably  will  not  be  able 
to  afford  any  above  this  price.  Don’t  make  the  mistake  of  screening  exactly  at  $500 
because  there  may  be  better  options  right  above  $500  in  which  you  may  be  able  to 
negotiate  or  get  discounts  which  will  make  them  affordable.  It  is  important  to  select 
screening  criteria  that  is  relatively  loose  so  not  to  exclude  alternatives  that  would  be  most 
preferred. 

When  there  are  too  few  alternatives,  associative  reasoning  can  both  help  and 
hinder  the  process.  The  reasoning  process  can  help  because  they  may  generate  ideas  that 
do  not  seem  at  first  to  be  relevant,  but  turn  out  to  be  useful.  However  it  can  also  hinder 
the  process  because  it  allows  you  to  quickly  build  a  “good  story”  to  why  you  already 
have  all  the  alternatives  you  need.  Therefore,  there  is  a  tendency  to  “rush  to  judgment” 
and  select  an  alternative  before  giving  careful  consideration  to  other  possibilities 
(Kirkwood,  1997).  There  are  several  methods  for  developing  more  alternatives  but  most 
of  them  center  on  using  the  existing  list  of  alternatives  and  creating  more  attractive 
alternatives  from  those. 

3.8  Step  7:  Score  the  Alternatives 
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Once  the  alternatives  and  the  SDVF’s  are  in  place  the  scoring  part  is  pretty  easy. 
It’s  simply  a  matter  of  determining  the  x-axis  value  and  then  reading  the  value  off  the  y- 
axis.  The  most  important  part  of  the  step  is  ensuring  that  the  x-axis  has  been  “clearly” 
defined.  You  want  to  ensure  that  if  someone  was  analyzing  your  model  with  the  same 
alternatives  10  years  from  now  that  they  would  come  to  the  same  conclusions. 

Years  ago  the  scoring  process  was  a  tedious  one  done  by  subject  matter  experts 
considering  each  alternative  for  a  particular  measure  before  advancing  to  the  next.  This 
allowed  SME’s  to  maintain  clarity  for  each  measure  definition  and  its  associated 
categories  along  the  x-axis  and  ensured  each  alternative  was  scored  consistently  (Jurk, 
2002)  .  Today  finding  the  overall  values  for  the  alternatives  using  the  value  functions  is 
pretty  simple  as  the  calculations  are  generally  done  by  an  electronic  spreadsheet  or 
special  program. 

3.9  Step  8:  Deterministic  Analysis 

Deterministic  analysis  is  simply  the  process  of  taking  the  score  of  the  alternative 
that  was  achieved  in  step  7  and  multiplying  it  times  the  weight  the  decision  maker 
decided  upon  for  the  specific  measure  in  step  5.  There  are  two  value  functions  that  are 
primarily  used,  the  additive  value  function  and  the  multiplicative  value  function.  The 
additive  value  function  is  the  simplest  and  easiest  to  use  and  is  commonly  used  among 
value  modelers.  The  additive  value  model  is  depicted  below  in  Equation  7 
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Equation  7 


where  w  represents  the  weight  of  the  particular  measure  and  v  represents  the  value  given 
of  the  particular  alternative  for  all  alternatives.  These  values  are  added  up  and  each 
alternative  is  given  a  score  from  0  to  100  based  on  how  it  scored  on  each  measure  in  the 
model.  At  this  point  there  is  a  list  of  alternatives  that  are  ranked  from  1  to  n  and 
sensitivity  analysis  can  begin. 

3.10  Sensitivity  Analysis 

Sensitivity  analysis  is  the  process  of  taking  the  ranked  list  of  alternatives  and 
determining  if  small  changes  in  weights  would  cause  the  rank  order  to  change.  During 
this  process  typically  the  weights  of  one  of  the  measures  are  changed  within  some 
specified  range  while  holding  the  weights  on  the  other  measures  constant.  Sensitivity 
analysis  can  also  be  completed  on  SDVF’s  but  it’s  really  not  a  feasible  technique  as  you 
don’t  see  a  great  deal  of  change  by  doing  this  (Weir,  2010).  This  process  shows  the  DM 
how  important  his  weights  are  and  if  they  changed  their  mind  on  what  was  important, 
which  alternative  would  be  the  most  attractive.  Sensitivity  analysis  can  be  performed  on 
the  local  or  global  weights. 

The  current  strategy  for  sensitivity  analysis  and  changing  weights  is  depicted 
below  in  Equation  8 


Equation  8 
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where  Wj  represents  all  changing  weights  in  the  sensitivity  analysis,  ws  represents  the 
weight  under  consideration  represents  all  changing  weights’  original  values  in  the 
first  model  and  m  represents  the  number  of  dependent  weights  (Weir,  2010).  This 
analysis  is  single  dimensional  and  only  allows  one  weight  manipulation  for  analysis. 

There  are  several  ways  to  do  sensitivity  analysis,  but  the  two  main  ways  that  are 
commonly  used  are  global  and  local  proportional.  Global  proportional  is  used  to 
determine  how  much  weight  would  have  to  be  taken  from  the  entire  model  to  change  the 
preferred  alternative.  This  method  is  mostly  used  when  there  is  one  DM  making  all  the 
decisions  about  the  weights.  If  there  is  one  DM  weights  at  the  top  of  the  hierarchy,  but 
the  branches  are  controlled  by  other  personnel,  then  local  proportional  weighting  is  used. 
This  method  allows  sensitivity  analysis  to  take  place  at  lower  levels  of  the  hierarchy 
without  changing  the  weights  on  the  top  values.  It  depends  on  what  type  of  analysis  is 
being  done  which  technique  would  be  best  to  use. 

3.11  Recommendations  and  Presentation 

Once  sensitivity  analysis  has  been  completed  it’s  time  to  present  the  DM  with  the 
results.  The  DM  may  or  may  not  have  a  strong  math  background  so  instead  of  boring 
them  with  information  on  how  the  results  were  attained,  get  straight  to  the  point  and  let 
them  know  what  their  best  alternatives  are.  This  is  also  an  opportunity  to  give  them  some 
insight  on  their  weight  sets  and  how  sensitivity  some  of  them  are.  This  information  will 
be  extremely  beneficial  and  enlightening,  especially  if  they  were  not  too  sure  on  their 
weights  in  the  first  place. 
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3.12  Summary 


VFT  is  a  ten  step  process  that  begins  with  determining  what  the  true  problem  is 
and  ends  with  briefing  the  recommendations  from  the  analysis.  Within  those  steps,  1-5 
require  plenty  of  DM  or  stakeholder  input  and  steps  6-10  are  done  at  the  discretion  of  the 
analyst.  The  overall  purpose  of  the  process  is  to  have  a  decision  making  process  that  is 
defendable,  repeatable  and  allows  sensitivity  analysis  to  identify  areas  where  a  small 
change  in  the  weights  can  change  the  desired  or  preferred  alternative. 

Using  these  steps,  a  real  world  example  will  be  performed  on  the  JSTARS  in 
chapter  4  to  determine  which  environments  maximize  the  GMTI  capability.  Through  this 
analysis,  it  will  illuminate  some  of  the  good  and  bad  elements  of  how  this  capability  is 
currently  being  used. 


Chapter  4  Results  and  Analysis 

In  chapter  3  a  great  deal  of  attention  was  taken  to  explain  the  ten  step  value 
focused  thinking  process.  In  this  chapter,  a  brief  explanation  of  the  specific  steps  that 
were  taken  for  this  particular  thesis  work  will  be  given.  The  majority  of  this  chapter  will 
focus  on  the  deterministic  and  sensitivity  analysis,  steps  8  and  9  of  the  value-focused 
thinking  process.  This  section  will  focus  mainly  on  how  and  why  the  preferred  alternative 
rose  to  the  top  and  others  did  not.  Additionally,  the  results  of  the  sensitivity  analysis  on 
the  local  and  global  weights  are  examined  to  see  how  changes  in  weights  would  influence 
the  ranking  of  the  most  preferred  alternative. 
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4.1  Problem  Identification 


The  sponsoring  agency  of  this  work  (DIA)  presented  the  problem  of  having  no 
way  to  model  or  measures  the  effectiveness  of  an  asset  with  the  GMTI  capability.  It  was 
decided  to  use  the  JSTARS  as  the  test  case  since  it  is  the  premier  GMTI  asset  in  the  AF 
inventory  even  though  there  are  other  assets  with  this  capability.  Specifically,  they 
wanted  to  know  “how  do  you  determine  the  effectiveness  of  GMTI  when  there  is  no 
amount  of  traffic  that  makes  this  capability  more  or  less  effective?”  As  stated  previously, 
they  currently  use  MOP’s  to  measure  their  effectiveness.  Unfortunately,  the  MOP’s  they 
use  do  not  translate  into  usable  information  when  trying  to  model  how  many  GMTI  assets 
are  required  or  how  well  they  are  doing  collectively  when  modeling  the  intelligence 
process. 

4.2  Creation  of  the  Hierarchy 

The  intent  was  to  use  intelligence  analyst  from  the  United  States  Central 
Command  as  the  subject  matter  experts,  since  the  aircraft  is  currently  deployed  in  its’ 
AOR.  However,  after  multiple  attempts  to  meet  and  subsequent  cancellations,  it  became 
obvious  that  there  was  a  need  to  use  an  alternative  subject  matter  expert  (SME).  The 
decision  was  made  to  use  the  men  and  women  of  the  1 16  ACW  as  the  SME’s. 
Specifically,  the  SME’s  consisted  of  Senior  Directors,  Surveillance  Officers,  Mission 
Crew  Commanders,  Deputy  Mission  Crew  Commanders,  and  Senior  Surveillance 
Mangers.  A  list  of  the  crew  duties  can  be  found  in  appendix  A  along  with  the  names  and 
duty  titles  in  appendix  D. 
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The  value  hierarchy  was  created  over  two  4  hours  periods  using  the  affinity 
diagram  method.  During  this  time  the  SME’s  named  all  the  values  that  were  key  in  the 
successful  implementation  of  GMTI.  After  some  lively  discussion,  grouping  and 
regrouping  they  came  up  with  the  hierarchy  that  is  depicted  in  Figure  11. 


Figure  11:  Values  of  GMTI  Hierarchy 

In  Table  6  below  are  definitions  of  what  each  value  means  and  how  each  value  affects  the 
effectiveness  of  GMTI. 
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Table  6:  Definition  of  Values 


Values 

Definitions 

Preparation 

Deals  with  the  pre-analysis  that  takes  place  prior  to  the 
engagement  beginning 

Aircraft 

Deals  with  things  that  can  be  directly  controlled  by  the 
crew  or  planners  of  the  missions. 

Targets 

Deals  with  things  that  directly  deal  with  the  environment 
of  the  targets 

Analysis 

Deals  with  the  during  mission  and  post  mission  analysis  of 
the  information  being  provided 

Battlefield 

Deals  with  things  that  are  not  target  related  and  concerns 
other  aircraft  and  locality  of  JSTARS 

4.3  Develop  Measures 

During  the  same  period,  the  SME’s  went  ahead  and  developed  the  evaluation 
measures  for  the  hierarchy.  If  you  recall,  measures  are  either  natural  or  constructed  and 
direct  or  proxy.  (See  section  3.4  if  you  need  a  reminder  of  what  this  means.)  In  Table  7  it 
shows  the  name  of  the  measure  and  its  definition.  This  information  was  used  to  build  the 
second  tier  of  the  value  hierarchy  structure  that  is  depicted  in  Figure  12. 
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Table  7:  Measure  Definitions 


Value 

Measure  Name 

Definition 

Preparation 

Intelligence 
Preparation  of  the 
Battlefield  (IPB) 

Categorical  (yes  or  no)  measure.  Where  the  crews  able 
to  get  in  country  before  the  engagement  began  to  get  an 
idea  of  what  the  steady-state  traffic  flow  was  like. 
JSTARS  specific. 

Aircraft 

Surveillance  Area 

Categorical  Measure  with  4  or  less  being  the  best  and  9 
or  more  being  the  worst.  The  trackers  onboard  the 
aircraft  can  only  track  about  4-15X15  areas  at  any  one 
time  with  a  high  degree  of  accuracy.  Once  the  tracking 
areas  get  larger  in  size  or  more  than  4  the  tracking 
accuracy  goes  down. 

Altitude 

Decreasing  Single  Dimensional  Value  Function  with 
10,000  being  the  worst  and  28,000  being  the  best.  The 
aircraft  has  an  optimal  AGL  altitude  that  maximizes 
radar  performance,  as  you  get  below  that  altitude  the 
radar  performance  degrades. 

Distance 

Decreasing  and  increasing  SDVF.  Is  the  distance  the 
aircraft  is  from  the  area  of  interest  the  optimal  distance 
for  radar  performance?  The  optimal  distance  is  the 
distance  located  in  the  Jane’s  manual.  Measure  is 
penalized  for  the  aircraft  for  being  too  close  &  too  far 
away. 

Targets 

Terrain 

Categorical  Measure  with  5  different  categories.  The 
categories  from  best  to  worst  are  Water,  Desert,  Light 

V egetation/Gras slands ,  Urban/Mountainous 

Environment,  and  Ice.  The  type  of  terrain  that  the 
aircraft  is  operating  in  plays  a  significant  role  on  how 
well  the  radar  performs. 

Weather 

Categorical  Measure  4  different  categories.  The 
categories  from  best  to  worst  are  Dry,  Light 

Precipitation,  Heavy  Precipitation,  and  Snow/Ice.  The 
weather  the  targets  are  operating  in  plays  a  major  role  in 
radar  performance. 

Type 

Categorical  Measure  with  5  different  categories.  The 
categories  from  best  to  worst  are  Large  Boats,  Tanks, 
Car/Truck,  Human,  and  Birds.  The  category  size  of  the 
target  helps  identify  the  target  easier;  therefore  this 
measure  will  be  defined  by  the  radar  cross  section  of  the 
target. 
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Number 

Increasing  SDVF  with  1  being  the  worst  and  15  or  more 
being  the  best.  The  total  number  of  targets  in  each  in  the 
particular  area  that  are  being  tracked  on  the  particular 
target  helps  great  increase  the  fidelity  of  the  targets. 

Analysis 

Positive  Identification 
(PID) 

Categorical  (yes  or  no)  measure.  Do  you  have  a 
“proper”  positive  identification  asset?  For  example  it 
would  not  be  a  proper  PID  asset  for  a  human  walking  to 
ID  a  vehicle  driving  down  the  road. 

Communication 

Categorical  (yes  or  no)  measure.  Are  you  directly 
working  with  an  agency  that  is  prosecuting  the 
particular  target? 

Feedback 

Categorical  (yes  or  no)  measure.  After  the  mission  is 
complete  are  you  getting  any  type  of  feedback  from  the 
agencies  that  you  are  supporting  on  how  helpful  the 
information  was  that  you  provided.  Also  what  you  can 
do  on  the  next  mission  to  enhance  the  value  of  the 
information  you  are  providing.  This  is  not  the  agency 
that  you  were  working  with  to  prosecute  the  target. 

Battlefield 

Location 

Categorical  (yes  or  no)  measure.  Is  the  orbit  that  is 
provided  in  the  best  location  to  see  the  particular  target. 

De-Confliction 

Categorical  (yes  or  no)  measure.  Has  the  airspace  been 
de-conflicted  so  that  the  radar  and  communications  are 
not  being  jammed  by  another  asset? 

It  is  extremely  important  to  note  that  an  independent  study  done  by  the  MITRE 
Corporation  showed  some  of  the  same  things  to  be  important  factors  when  attempting  to 
optimize  the  GMTI  capability.  The  factors  they  found  important  in  optimizing  GMTI 
were  mission,  target,  environment  and  sensor  selection  (Bonaceto,  Mooers,  Theophanis, 
&  Wrick,  2010).  Target  and  environment  were  already  captured  in  the  model.  Mission  is 
captured  as  well  as  the  model  in  Figure  12  is  the  High  Value  Target  model,  when 
prosecuting  a  forensics  only  mission  the  “Analysis”  value  would  be  deleted  along  with 
everything  beneath  it.  The  final  thing  they  thought  to  be  important,  sensor  selection,  is 
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not  a  factor  in  this  study  as  there  is  only  one  to  choose.  Thus  there  independent  study 
helped  validate  the  working  model. 


GMTI 

Effectiveness 


I - 1 - 1 - 1 - 1 - 1 


Preparation 

1 

Aircraft 

1 

Targets 

Analysis 

Battlefield 

Figure  12:  GMTI  Hierarchy  with  Measures 


4.4  Create  Value  Functions 

Armed  with  the  measures  and  their  definitions  the  SME’s  next  proceeded  to 
create  the  value  functions  for  each  measure.  There  were  13  measures  and  of  those  13,  ten 
were  given  piecewise  linear  or  categorical  SDVF’s  and  the  other  three  were  given 
exponential  SDVF’s.  The  10  categorical  measures  were  designated  as  such  because  there 
were  only  a  small  amount  of  vales  that  each  category  could  possibly  be.  The  measures 
distance,  altitude,  and  number  were  all  given  exponential  SDVF’s  because  they  could 
take  on  10  or  more  values.  All  of  the  SDVF’s  are  located  in  appendix  3.  The  only  thing 
remarkable  about  any  of  the  functions  was  the  one  for  distance.  Since  this  measure 


65 


penalized  the  aircraft  for  being  too  close  and  too  far  away,  there  had  to  be  two 
exponential  functions  created.  In  figure  13,  the  exponential  SDVF  “Distance  I”  measures 
when  the  aircraft  is  between  0  to  50  miles. 


Figure  13:  Distance  Measure  I  SDVF 


The  second  exponential  SDVF  “Distance  II”  in  Figure  14  measures  when  the  aircraft  is 
between  60  miles  away  or  greater.  Anything  between  50  and  60  automatically  gets  a 
score  of  1.  Since  the  software  (Hierarchy  Builder)  only  allows  one  SDVF  per  measure,  if 
the  aircraft  distance  was  located  in  “distance  II”  then  it  required  the  user  to  interpret  the 
data  and  place  it  in  “distance  I”  For  example,  if  the  aircraft  was  70  miles  away  the  user 
would  have  to  interpret  where  70  was  located  on  the  y-axis  in  Figure  14  (.76)  then  take 
that  information  and  put  it  into  the  y-axis  in  “distance  I”  in  Figure  13  and  determine  the 
x-axis  value  (40)  which  is  the  number  that  would  go  into  the  model. 
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Distance  II 


Figure  14:  Distance  Measure  II SDVF 

4.5  Weight  the  Hierarchy 

The  weighting  of  the  hierarchy  was  done  using  the  direct  assessment  method. 
Each  SME  was  explained  how  the  weighting  process  worked.  After,  they  were  all  given  a 
sheet  with  the  entire  hierarchy  and  told  to  independently  determine  what  they  thought  the 
weights  should  be  for  each  measure  &  each  value.  Once  complete,  all  weights  were  put 
on  the  board.  Any  weights  that  were  significantly  different  were  discussed  and  a  re¬ 
reweighting  process  was  done  on  the  measures  independently.  Remarkably,  the  weights 
on  the  first  try  were  very  similar  and  there  were  only  a  few  differences  that  needed  to  be 
discussed  and  reweighed.  The  final  weighted  Hierarchy  is  located  below  in  Figure  15. 
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GMTI  Effectiveness 


i - 1 - 1 - 1 - 1 - 1 


H  H 


Figure  15:  Final  Weighted  Hierarchy 


4.6  Alternative  Generation 

The  alternative  generation  stage  required  some  thought  and  resourcefulness  to  get 
the  alternatives  down  to  a  manageable  number.  At  first  glance  it  was  thought  to 
enumerate  every  possible  combination.  With  13  measures  and  10  of  them  being 
categorical,  enumerating  the  10  categorical  would  be  25,600  alternatives.  If  the  3 
exponential  SDVF’s  that  can  take  on  an  infinite  amount  of  possibilities  were  added,  there 
would  be  an  intractable  number  of  alternatives.  Even  if  the  exponential  SDVF’s  were 
broken  into  4  different  quadrants  there  would  still  be  1.6  million  possible  alternatives. 

To  arrive  at  a  manageable  number  of  alternatives  it  was  decided  to  use  a 
preponderance  of  the  weight  to  derive  the  alternatives.  Using  the  measures  IPB,  Sur  Area, 
Terrain,  Weather,  Type,  Location,  De-confliction,  and  Communication  comprised  of 
83.3%  of  the  total  hierarchy  as  seen  in  Table  8.  Any  alternative  that  rose  to  the  top  of  this 
modified  hierarchy  will  also  be  in  the  top  of  the  overall  hierarchy.  For  example,  the 
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measure  “Weather”  which  was  included  in  the  modified  hierarchy  was  enumerated  by  its 
four  different  categories  of  ice/scow,  heavy  precipitation,  light  precipitation,  and  dry 
where  as  the  measure  “Number”  was  simple  given  the  value  that  would  achieve  a  score 
of  one  in  that  measure  which  was  15.  Table  8  depicts  the  global  weight  of  each  measure. 
The  measures  that  are  highlighted  are  the  measures  that  were  optimized  to  their  max 
performance  during  the  scoring  of  the  alternatives.  These  measures  were  chosen  not  to  be 
included  because  they  had  the  lowest  global  weights  and  because  many  of  them  are 
controlled  by  the  planners  and  crews.  By  doing  this,  each  alternative  score  was 
artificially  inflated  17%  before  the  scoring  process  ever  began. 


Table  8:  Measure  Order  by  Global  Weight 


Measure  Rankins 

Measure 

Global  Weight 

Terrain 

0.1665 

Location 

0.16 

Surface  Area 

0.13 

Weather 

0.111 

Communication 

0.09 

IPB 

0.08 

Target  Type 

0.0555 

Distance 

0.04 

DeConfliction 

0.04 

Number 

0.037 

Altitude 

0.03 

PID 

0.03 

Feedback 

0.03 

Using  the  8  remaining  measures  (all  categorical)  9600  alternatives  were 
generated.  Even  though  this  number  was  far  less  than  the  1.6  million  or  more  that  could 
have  been  generated,  it  was  still  far  too  many.  To  get  an  acceptable  number  of 
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alternatives,  the  9600  alternatives  were  input  into  the  model  and  once  they  were  ranked 
the  top,  middle,  and  bottom  400  alternatives  would  be  used  for  analysis.  1200  alternatives 
still  made  it  difficult  to  perform  good  analysis,  so  those  1200  were  broken  down  even 
further.  Using  the  weather  measure,  each  or  the  3  groups  were  broken  down  into  4 
categories  of  dry ,  heavy  precipitation ,  light  precipitation ,  and  snow/  ice.  Within  the 
groups  they  were  separated  into  4  groups  of  100.  Finally,  a  random  draw  was  taken  from 
each  of  the  16  groups  to  come  up  with  the  alternatives  used  for  analysis. 


Table  9:  Top  400  Alternatives  Broken  into  Weather  Categories 


Top  400  Alternatives 

Tod  100 

101-200 

201-300 

301-400 

Dry 

56 

50 

45 

44 

Heavy  Precip 

8 

18 

16 

20 

Light  Precip 

36 

32 

31 

31 

Snow/Ice 

0 

0 

8 

5 

For  example,  Table  9  represents  the  break-out  of  the  top  400  alternatives.  One  alternative 
was  randomly  selected  from  each  group  providing  14  alternatives  for  analysis  since  there 
is  no  snow/ice  alternative  in  the  top  200  alternatives.  Completing  this  exercise  for  the 
middle  and  bottom  alternatives  produced  46  alternatives  that  could  be  easily  manipulated 
to  conduct  deterministic  and  sensitivity  analysis.  There  was  also  some  analysis  done  on 
any  alternative  that  scored  75%  or  better  in  the  model  of  which  there  was  1758. 

4.7  Score  the  Alternatives 

Once  a  manageable  number  of  alternatives  was  reached,  the  alternatives  were 
rescored  and  used  for  analysis.  The  scores  of  the  46  alternatives  used  are  located  in  Table 
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11.  The  measures  that  were  not  used  in  the  final  scoring  of  the  alternatives  are  not 
depicted  in  the  Table  10  nor  are  they  depicted  in  Figure  16,  the  graphical  depiction  of  the 
scored  measures. 
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Table  10:  Alternative  Scores 


Possible  Score 

0.1665 

0.16 

0.13 

0.111 

0.09 

0.08 

0.0555 

0.04 

1 

Measure 

Terrain 

Location 

Sur  Areas 

Weather 

Comms 

IPB 

Type 

Deconflict 

Score 

light  2244  0.969 

0.1582 

0.16 

0.130 

0.089 

0.09 

0.08 

0.0555 

0.04 

0.9695 

Dry  2242  0.959 

0.1582 

0.16 

0.098 

0.111 

0.09 

0.08 

0.0555 

0.04 

0.9592 

Heavy2184  0.922 

0.1582 

0.16 

0.130 

0.050 

0.09 

0.08 

0.0472 

0.04 

0.9223 

light  1013  0.914 

0.1582 

0.16 

0.117 

0.089 

0.09 

0.08 

0.0527 

0.00 

0.9137 

Heavy2218  0.904 

0.1665 

0.16 

0.098 

0.050 

0.09 

0.08 

0.0527 

0.04 

0.9037 

Dry  929  0.892 

0.1665 

0.16 

0.117 

0.111 

0.09 

0.08 

0.0000 

0.00 

0.8915 

light  1650  0.888 

0.1665 

0.16 

0.130 

0.089 

0.00 

0.08 

0.0555 

0.04 

0.8878 

Hea  vyl013  0.875 

0.1582 

0.16 

0.117 

0.050 

0.09 

0.08 

0.0527 

0.00 

0.8749 

Snow2184  0.872 

0.1582 

0.16 

0.130 

0.000 

0.09 

0.08 

0.0472 

0.04 

0.8724 

Dry  981  0.872 

0.1582 

0.16 

0.059 

0.111 

0.09 

0.08 

0.0472 

0.00 

0.8719 

Snow2189  0.868 

0.1665 

0.16 

0.117 

0.000 

0.09 

0.08 

0.0472 

0.04 

0.8677 

Hea  vyl048  0.866 

0.1665 

0.16 

0.098 

0.050 

0.09 

0.08 

0.0555 

0.00 

0.8665 

light  1036  0.864 

0.1249 

0.16 

0.098 

0.089 

0.09 

0.08 

0.0555 

0.00 

0.8637 

Dry  1582  0.861 

0.1582 

0.16 

0.098 

0.111 

0.00 

0.08 

0.0472 

0.04 

0.8609 

Hea  vy2289  0.627 

0.0333 

0.16 

0.059 

0.050 

0.09 

0.00 

0.0278 

0.04 

0.6265 

light  2288  0.626 

0.0333 

0.16 

0.020 

0.089 

0.09 

0.00 

0.0278 

0.04 

0.6264 

Snow432  0.626 

0.0333 

0.16 

0.130 

0.000 

0.00 

0.08 

0.0555 

0.00 

0.6258 

Dry  398  0.624 

0.0333 

0.16 

0.020 

0.111 

0.00 

0.08 

0.0527 

0.00 

0.6235 

Dry  2006  0.622 

0.1665 

0.00 

0.020 

0.111 

0.09 

0.00 

0.0278 

0.04 

0.6218 

Hea  vy365  0.621 

0.0000 

0.16 

0.117 

0.050 

0.00 

0.08 

0.0472 

0.00 

0.6211 

light  856  0.621 

0.1249 

0.00 

0.098 

0.089 

0.09 

0.00 

0.0527 

0.00 

0.6209 

Snowl625  0.620 

0.0000 

0.16 

0.117 

0.000 

0.00 

0.08 

0.0555 

0.04 

0.6195 

light  1924  0.619 

0.0000 

0.00 

0.098 

0.089 

0.09 

0.08 

0.0555 

0.04 

0.6188 

Hea  vyl842  0.618 

0.0333 

0.00 

0.130 

0.050 

0.09 

0.08 

0.0278 

0.04 

0.6180 

Dry  368  0.618 

0.0333 

0.16 

0.020 

0.111 

0.00 

0.08 

0.0472 

0.00 

0.6180 

Snow2290  0.616 

0.0333 

0.16 

0.098 

0.000 

0.09 

0.00 

0.0278 

0.04 

0.6156 

Heavy2131  0.615 

0.0000 

0.16 

0.000 

0.050 

0.09 

0.08 

0.0278 

0.04 

0.6147 

Dry  1221  0.615 

0.1582 

0.00 

0.059 

0.111 

0.00 

0.08 

0.0000 

0.04 

0.6147 

light  136  0.614 

0.1249 

0.00 

0.098 

0.089 

0.00 

0.08 

0.0555 

0.00 

0.6137 

Snowl905  0.613 

0.1249 

0.00 

0.059 

0.000 

0.09 

0.08 

0.0527 

0.04 

0.6131 

Snow229  0.372 

0.1582 

0.00 

0.000 

0.000 

0.00 

0.00 

0.0472 

0.00 

0.3724 

light  273  0.370 

0.0000 

0.00 

0.059 

0.089 

0.00 

0.00 

0.0555 

0.00 

0.3698 

Dry  247  0.364 

0.0333 

0.00 

0.000 

0.111 

0.00 

0.00 

0.0527 

0.00 

0.3640 

Hea  vy758  0.360 

0.0333 

0.00 

0.020 

0.050 

0.09 

0.00 

0.0000 

0.00 

0.3598 

Snow481  0.355 

0.0000 

0.16 

0.000 

0.000 

0.00 

0.00 

0.0278 

0.00 

0.3548 

Hea  vyl383  0.343 

0.0000 

0.00 

0.059 

0.050 

0.00 

0.00 

0.0278 

0.04 

0.3432 

light  247  0.342 

0.0333 

0.00 

0.000 

0.089 

0.00 

0.00 

0.0527 

0.00 

0.3418 

Dry  1352  0.338 

0.0000 

0.00 

0.020 

0.111 

0.00 

0.00 

0.0000 

0.04 

0.3375 

Snowl981  0.325 

0.0000 

0.00 

0.000 

0.000 

0.09 

0.00 

0.0278 

0.04 

0.3248 

Hea  vyl412  0.324 

0.0000 

0.00 

0.020 

0.050 

0.00 

0.00 

0.0472 

0.04 

0.3236 

light  187  0.317 

0.0333 

0.00 

0.000 

0.089 

0.00 

0.00 

0.0278 

0.00 

0.3169 

Dry  157  0.311 

0.0333 

0.00 

0.000 

0.111 

0.00 

0.00 

0.0000 

0.00 

0.3113 

Dry  151  0.278 

0.0000 

0.00 

0.000 

0.111 

0.00 

0.00 

0.0000 

0.00 

0.2780 

light  152  0.275 

0.0000 

0.00 

0.020 

0.089 

0.00 

0.00 

0.0000 

0.00 

0.2753 

Snowl353  0.266 

0.0000 

0.00 

0.059 

0.000 

0.00 

0.00 

0.0000 

0.04 

0.2655 

Hea  vyl82  0.264 

0.0000 

0.00 

0.020 

0.050 

0.00 

0.00 

0.0278 

0.00 

0.2642 
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Dry  2242 

0.959 

light  1013 

0.914 

Dry  929 

0.892 

Heavy 1013 

0.875 

Dry  981 

0.872 

Heavyl048 

0.866 

Dry  1582 

0.861 

light  2288 

0.626 

Dry  398 

0.624 

Heavy365 

0.621 

Snowl625 

0.620 

Heavyl842 

0.618 

Snow2290 

0.616 

Dry  1221 

0.615 

Snowl905 

0.613 

light  273 

0.370 

Heavy758 

0.360 

Heavyl383 

0.343 

Dry  1352 

0.338 

Heavy 1412 

0.324 

Dry  157 

0.311 

light  152 

0.275 

Heavyl82 

0.264 

□  Terrain  BLocation  DSurAreas  BWeather  DComms  DIPB  DType 


□  Deconflict 


Figure  16:  Breakout  of  Alternative  Scores  by  Measure 

4.8  Deterministic  Analysis 

In  beginning  the  deterministic  analysis,  it  was  pretty  evident  that  the  alternatives 
that  performed  well  in  terrain,  location,  surface  areas  and  weather  would  also  perform 
well  in  this  model  as  these  four  measures  account  for  56.75%  of  the  total  model.  In 
Figure  16  the  measures  are  arranged  in  order  of  their  weight.  Here  you  can  visually  see 
that  the  top  alternatives  are  dominated  by  alternatives  that  perform  well  in  these 
categories. 
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Before  the  alternatives  were  narrowed  down  there  were  some  observations  that 


need  to  be  mentioned  about  the  9600  alternative  set.  Among  the  top  400  alternatives  the 
top  10  all  included  alternatives  that  included  dry  weather.  The  top  alternative  that 
included  heavy  precipitation  was  not  observed  until  #54  and  there  was  no  alternative  that 
included  snow/ice  until  #209.  The  assumption  was  made  that  any  mission  that  scored 
75%  or  above  in  the  model  would  be  considered  an  effective  mission  and  as  stated  before 
there  were  1758  combinations  of  missions  that  meet  this  criteria. 


Table  11:  Alternatives  That  Met  the  75%  Cutoff  Score  by  Weather 


Snow/Ice 

181 

0.103 

Light 

545 

0.310 

Heavy 

360 

0.205 

Dry 

672 

0.382 

Among  the  1758  alternatives,  69%  include  light  precipitation  or  dry  weather  and  only 
10%  are  in  snow/ice  conditions  as  seen  in  Table  11. 

In  order  to  conduct  a  thorough  analysis  of  the  data  the  data  was  broken  down  by 
measure  in  order  of  the  weight  of  the  measure.  The  first  measure  analyzed  was  terrain. 


Table  12:  Top,  Middle  &  Bottom  Alternatives  400  broken  out  by  Terrain 


Terrain 

Top 

Middle 

Bottom 

Category 

Number 

Category 

Number 

Category 

Number 

Water 

170 

Water 

76 

Water 

3 

Desert 

149 

Desert 

81 

Desert 

6 

Light  Veg 

76 

Light  Veg 

84 

Light  Veg 

18 

Urban/Mount 

3 

Urban/Mount 

83 

Urban/Mount 

139 

Ice 

0 

Ice 

75 

Ice 

234 
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Terrain  is  the  measure  that  has  the  most  weight  in  the  entire  model  atl6.5%.  Of  the  1758 
alternatives  that  scored  75%  or  better  in  the  model,  1599  included  water ,  desert ,  or  light 
vegetation  as  its  terrain  option.  Table  12  shows  the  top,  middle  and  bottom  400 
alternatives  broken  out  by  the  Terrain  measure.  In  the  top  400  there  are  no  alternatives 
that  include  ice  as  its  terrain  option  and  there  were  only  3  that  include 
urban/mountainous  terrain.  In  the  top  1758  there  were  only  159  or  9%  that  contained  ice 
or  urban/mountainous  terrain  as  its  environments.  Conversely,  even  though  water 
provides  the  best  conditions  to  track,  there  were  3  alternatives  that  put  tracking  in  a  water 
environment  in  the  bottom  400  alternatives  of  the  model.  The  bottom  400  alternatives  are 
dominated  by  alternatives  that  consist  of  tracking  in  urban/mountainous  or  ice  terrain.  In 
looking  at  Table  12,  373  of  the  400  or  93.25%  of  the  alternatives  consist  of 
urban/mountainous  or  ice  terrain  environments.  The  alternatives  in  the  middle  of  the 
model  show  an  even  distribution  of  each  the  terrain  categories.  In  looking  at  the  data,  this 
will  be  a  common  theme  for  each  of  the  measures. 

The  next  measure  that  was  analyzed  was  location  with  16%  of  the  model  weight. 
The  raw  data  in  Table  13  shows  that  there  are  no  alternatives  in  the  top  400  that  do  not 
score  well  in  this  category.  Furthermore,  of  the  alternatives  that  meet  the  75%  cut  off, 
there  are  only  130  that  do  not  score  a  yes  in  this  category.  That  means  93%  of  the 
alternatives  scored  well  in  this  category.  This  demonstrates  that  mission  success  is  highly 
dependent  on  the  aircraft  being  in  the  correct  location. 
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Table  13:  Top,  Middle  and  Bottom  400  broken  out  by  the  Location  Measure 


Location 

Top 

Mic 

die 

Bottom 

Category 

Number 

Category 

Number 

Category 

Number 

Yes 

400 

Yes 

192 

Yes 

5 

No 

0 

No 

207 

No 

395 

The  number  of  areas  each  tracker  has  to  monitor  was  the  next  measure  with  13% 
of  the  overall  model.  Figure  17  illustrates  the  400  alternatives  in  this  category.  The 
numbers  four,  five ,  and  six  consisted  of  84%  of  the  alternatives.  Even  when  the 
alternative  set  was  expanded  out  to  top  the  18%  of  the  9600  alternatives,  Table  14  shows 
that  categories  four,  five  and  six  consisted  of  almost  75%  of  the  alternatives.  This  shows 
that  the  correct  location  is  a  critical  component  to  mission  success. 


Figure  17:  Top  400  Surface  Area  Alternatives 
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Table  14:  Top  1758  Surface  Area  Alternatives 


Surface  Area 

Category 

Number 

Four 

471 

Five 

451 

Six 

362 

Seven 

236 

Eight 

136 

Nine 

102 

Figure  18  clearly  shows  that  when  the  trackers  are  over  tasked  and  have  to  look  at  seven 
or  more  areas  that  these  alternatives  perform  poorly  as  342  of  the  400  alternatives  are 
found  in  categories  where  they  are  tracking  in  more  than  seven  areas.  Therefore,  keeping 
the  numbers  of  areas  they  track  in  the  area  of  six  or  less  dramatically  increase  the 
probability  of  having  a  successful  mission. 


Figure  18:  Bottom  400  Surface  Area  Alternatives 

The  weather  measure  completes  the  measures  with  double  digits  percentages  as  it 
comes  in  at  11.1%  of  the  overall  model.  Dry  weather  and  light  precipitation  dominated 
the  majority  of  alternatives  in  the  top  400  consisting  of  over  82%  of  the  total  alternatives 


77 


(see  Figure  19).  There  were  only  13  snow  /ice  alternatives  in  the  top  400,  less  than  4%  of 
the  total,  and  none  in  the  top  200.  Just  as  the  top  alternatives  were  dominated  by  two 
categories,  the  bottom  was  dominated  by  heavy  precipitation  and  snow/ice  with  a  little 
over  81%  consisting  of  these  two  weather  conditions  as  seen  in  Figure  20.  However, 
unlike  the  top  400  alternatives  in  this  measure  2  of  the  bottom  100  were  Dry  alternatives 
and  10  of  the  bottom  200.  This  provided  clear  evidence  that  even  in  the  best  tracking 
conditions  weather  wise,  there  could  still  be  poor  mission  results. 


■  Dry 

■  Heavy  Precip 

■  Light  Precip 

■  Snow/Ice 


Figure  19:  Top  400  Weather  Alternatives 
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■  Dry 

■  Heavy  Precip 

■  Light  Precip 

■  Snow/Ice 


Figure  20:  Bottom  400  Weather  Alternatives 


When  it  came  to  communication ,  9%  of  the  overall  model,  the  data  remained 
pretty  consistent.  Table  15  revealed  that  of  the  top  400  alternatives  89%  of  the 
alternatives  had  a  yes  response  in  communication  measure.  Even  when  expanded  out  to 
the  75%  cut  off  score,  74%  of  the  alternatives  still  maintained  a  yes  response  in  this 
particular  measure.  When  looking  at  the  bottom  of  the  alternative  list,  only  15%  of  those 
alternatives  had  a  yes  response  in  this  measure  which  is  pretty  consistent  with  the  top 
alternatives  with  a  small  difference  of  4%. 
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Table  15:  Top,  Middle  &  Bottom  400  Communication  Measure  Data 


Communication 

Top 

Mic 

die 

Bottom 

Category 

Number 

Category 

Number 

Category 

Number 

Yes 

355 

Yes 

207 

Yes 

61 

No 

45 

No 

192 

No 

339 

Figure  21:  Top  1758  Alternatives  in  Communication  Measure 


Like  the  communications  measure,  Intelligence  Preparation  of  the  Battle  Space 
remained  fairly  consistent  as  well.  Referencing  Table  16,  there  were  just  as  many  no 
responses  in  the  top  alternatives  as  there  were  yes  in  the  bottom  alternatives. 

Additionally,  taking  a  look  at  the  middle  responses  they  had  virtually  the  same  amount  of 
no  and  yes  responses  which  made  this  particular  measure  linear.  This  indicated  that 
among  the  alternatives,  the  amount  of  yes  to  no  responses  changed  equally  from  the  best 
to  the  worst  alternatives. 
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Table  16:  IPB  Measure  Data 


Intelliegnce  Preparation  of  the  Battle  Space 

Top 

Mic 

die 

Bottom 

Category 

Number 

Category 

Number 

Category 

Number 

Yes 

337 

Yes 

198 

Yes 

61 

No 

61 

No 

201 

No 

339 

The  final  measure  that  was  used  to  analyze  the  data  was  the  type  of  target  they 
were  tracking  which  garnered  5.55%  of  the  overall  model.  This  measure  showed  in 
Figure  22  that  the  top  alternatives  did  best  when  tracking  the  best  targets,  and  in  Figure 
24  the  bottom  alternatives  did  worst  when  tracking  the  worst  targets.  However,  it  also 
showed  that  these  obstacles  could  be  overcome  for  any  target  type  if  provided  with  the 
right  mix  of  the  other  measures.  Figure  23  shows  that  there  is  no  consistency  in  the 
middle  alternatives  on  which  target  type  is  more  preferred. 


Bird  Human  Car/Truck  Tank  Large 

Boat 


Figure  22:  Top  400  Target  Type  Alternatives 
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Figure  23:  Middle  400  Target  Type  Alternatives 


Figure  24:  Bottom  400  Target  Type  Alternatives 


4.9  Sensitivity  Analysis 
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As  a  reminder,  the  sensitivity  analysis  phase  deals  with  determining  if  small 
changes  in  the  weights  would  cause  the  user  to  make  a  different  decision.  There  are  two 
ways  to  do  sensitivity  analysis,  global  and  local  sensitivity  analysis.  Since  many  of  the 
alternatives  were  deleted  so  to  achieve  a  manageable  alternative  set,  it  was  impossible  to 
do  true  sensitivity  analysis.  Therefore  the  focus  was  on  showing  examples  of  sensitive 
and  non-sensitive  weights  and  or  measures. 

Figure  25  shows  an  example  of  a  non-sensitive  measure.  In  this  example,  the 


alternative  Heavy  1048  deterministically  dominates  all  other  alternatives. 


Sensitivity  Analysis  for  Number 


- Dry  1582 

- Dry  2242 

Dry  929 

- Dry  981 

Heavyl013 
Heavyl048 
Heavy2184 
Heavy2218 
light  1013 

- light  1036 

light  1650 

- light  2244 

Snow2184 

Snow2189 

Snow229 


Figure  25:  Non  Sensitive  Measure  (Target  Type) 
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At  no  point,  no  matter  what  the  value  of  the  weight  of  the  measure,  will  any  other 
alternative  perform  better  than  Heavy  1048.  An  example  of  a  sensitive  measure  can  be 
seen  in  Figure  26.  In  this  example  at  the  current  global  weight  of  1 1.1%  heavy  1048  is  the 
clear  #1  choice  for  this  measure.  However,  if  the  global  weight  of  this  measure  changed 
to  18%  or  greater,  the  DM  would  make  a  different  decision  and  chose  alternative  Dry 
2242  as  its  preferred  alternative.  This  is  a  global  analysis,  which  means  that  DM  would 
have  to  convert  7%  of  the  entire  model  weight  to  weather  for  the  preferred  alternative  to 


change. 


If  the  DM  is  happy  with  their  weights  on  the  values,  it  may  be  better  to  do  local 
sensitivity  analysis.  Figure  27  shows  an  example  of  local  sensitivity  analysis  on  the  same 
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measure  weather.  In  this  example,  the  blue  arrow  shows  that  the  DM  would  have  to  give 
over  half  of  the  weight  in  the  value  targets  to  change  the  preferred  alternative  which 
would  only  leave  .48  percent  for  the  three  remaining  measures.  So  while  this  measure 
seemed  sensitive  under  global  sensitivity  analysis,  under  local  sensitivity  analysis,  this 
measure  is  not  sensitive  at  all.  If  the  DM  did  a  decent  job  in  determining  the  weights 
during  the  building  of  the  hierarchy,  it  would  seem  infeasible  that  they  would  change  any 


local  weight  by  more  than  70%  of  its  original  weight. 


Sensitivity  Analysis  for  Weather 


- Dry  1582 

- Dry  2242 

Dry  929 

- Dry  981 

Heavyl013 
Heavyl048 
Heavy2184 
Heavy2218 
light  1013 
light  1036 
light  1650 
light  2244 
Snow2184 


Figure  27:  Local  Sensitivity  Analysis  (Weather) 


It  is  impossible  to  determine  the  true  sensitivity  of  this  model  without  all  the 
alternatives  inserted.  Based  on  the  46  alternatives  that  was  chosen,  this  model  is  pretty 
insensitive  to  the  weights.  However,  it  has  been  shown  that  small  changes  in  weights  can 
change  the  preferred  alternative.  If  in  a  situation  where  a  small  change  does  change  the 
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preferred  alternative,  it  is  best  to  notify  the  DM  as  they  may  want  to  redefine  their 
objectives  or  include  other  objectives. 

4.10  Summary 

Chapter  4  reviewed  the  steps  that  were  taken  in  creating  the  JSTARS  GMTI  value 
model  and  the  results  of  the  model  including  sensitivity  analysis.  The  deterministic 
analysis  showed  for  example  that  there  are  some  environments  that  GMTI  does  very  well 
in  such  as  water,  desert  and  light  vegetation,  but  other  environments  such  as  mountainous 
or  urban  or  icy  terrain  it  can  be  extremely  difficult,  but  not  impossible  to  overcome  these 
obstacles  and  achieve  a  mission  that  receives  a  score  of  75%  or  better.  In  the  final  section 
of  this  chapter  the  focus  was  on  sensitivity  analysis  and  how  to  determine  when  a 
measure  is  sensitive  using  local  and  global  sensitivity  analysis. 


Chapter  5  Findings  and  Conclusions 

Chapter  5  is  the  culmination  of  the  thesis  effort.  It  draws  conclusions  to  the 
application  of  the  value-focused  thinking  model  on  JSTARS  and  GMTI.  Here  the  focus 
the  overall  conclusions  of  this  particular  study,  recommendations  to  increase  the 
likelihood  of  having  a  successful  mission  and  some  future  research  that  could  or  should 
be  done  in  this  area. 

5.1  Study  Conclusions 
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One  of  the  first  observations  discovered  during  this  research  was  that  using  the 
measures  that  the  crew  or  planners  have  direct  control  over;  they  can  only  achieve  a  score 
of  .47.  Table  17  shows  the  controllable  and  uncontrollable  measures.  Since  the  score  of 
the  controllable  measures  is  so  low,  it  is  imperative  the  crew  and  planners  spend  the 
appropriate  amount  of  time  studying  the  terrain,  location,  and  weather  of  the  planned  area 
to  ensure  that  they  can  reach  an  acceptable  mission  score. 


Table  17:  GMTI  Measures:  Controllable  &  Uncontrollable 


GMTI  Measures 

Controllable  (.47) 

Uncontrollable  (.53) 

Surface  Areas  (.13) 

Terrain  (.1665) 

Communications  (.09) 

Location  (.16) 

IPB  (.08) 

Weather  (.111) 

Distance  (.04) 

Type  (.0555) 

De-Confliction  (.04) 

Number  (.037) 

Altitude  (.03) 

Positive  Identification  (.03) 

Feedback  (.03) 

The  next  conclusion  pertained  to  the  terrain  measure.  It  was  very  hard  and  almost 
impossible  to  have  a  measure  that  falls  in  the  mountainous  or  urban  or  ice  terrain 
categories  to  do  well  in  this  model.  Only  3  mountainous  or  urban  terrain  alternatives 
were  in  the  top  400  and  no  ice  alternatives.  Even  when  using  the  top  1758  there  were 
only  9%  of  the  alternatives  that  include  these  categories.  Remember  that  each  alternative 
had  an  artificial  17%  inflation  which  means  there  could  possibly  have  been  even  less  than 
9%  in  the  top  alternatives. 

It  was  also  impossible  to  score  well  in  the  model  if  the  aircraft  was  not  in  the 
correct  location.  400  of  the  top  400  all  received  a  yes  response  when  determining  if  they 
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were  in  the  correct  location.  When  it  came  to  the  amount  of  surveillance  areas  the 
trackers  had  to  monitor,  four,  five,  or  six  dominated  the  top  alternatives.  Once  they  got 
above  six,  the  possibility  of  scoring  well  in  the  model  decreased  dramatically. 

Most  of  the  top  alternatives  did  well  in  communications  and  Intelligence 
Preparation  of  the  Battle  Space,  but  the  data  showed  that  even  if  they  did  not;  either  of 
these  measures  could  be  overcome.  Communication  had  44  no  response  in  the  top  400, 
but  also  had  61  yes  responses  in  the  bottom  400.  Additionally,  IPB  had  61  no  responses 
in  the  top  400  and  74  yes  responses  in  the  bottom  400.  This  data  indicated  that  an 
alternative  could  score  well  in  these  measure  and  easily  be  in  the  top  or  bottom  of  the 
alternatives.  It  is  important  to  note,  that  even  if  IPB  was  not  accomplished  prior  to 
arriving  in  theater,  if  the  aircraft  remains  operational  in  theater  long  enough,  at  some 
point  IPB  can  be  considered  to  be  accomplished. 

The  final  conclusion  that  was  gleaned  from  this  research  was  about  the  targets. 
While  the  best  targets  such  as  boats  and  tanks  rise  to  the  top,  there  is  no  huge  disparity  in 
tracking  the  other  targets.  Tanks  and  large  boats  can  only  be  tracked  1.3  to  1  better  than 
humans,  cars  and  boats.  Out  of  the  top  400  alternatives,  208  consisted  of  tanks  and  large 
boats,  and  155  consisted  of  humans,  cars  and  trucks. 

5.2  Current  Operational  Data 

Since  JSTARS  in  currently  operating  in  the  Afghanistan  AOR,  this  model  was 
presented  to  the  crews  to  see  how  their  current  missions  would  score.  While  the  true 
score  is  classified,  it  can  be  seen  in  Figure  28  that  they  are  operating  in  the  middle  scores 
of  the  model  and  below  the  75%  that  was  assumed  to  be  an  acceptable  mission  score. 
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Some  of  the  issues  that  lowered  their  score  were  altitude,  terrain,  type  of  target,  positive 


identification,  communications,  and  feedback. 
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Figure  28:  GMTI  Effectiveness  Alternative  Scores 


5.3  Recommendations 

There  were  several  recommendations  that  came  out  of  this  analysis.  The  first  was 
to  always  have  the  trackers  monitoring  six  areas  or  less.  The  scores  of  the  mission  rose 
considerably  when  monitoring  in  six  or  less.  The  can  do  this  in  one  of  two  ways.  The  first 
is  by  adding  an  additional  tracker  to  the  crew  and  have  them  take  over  one  of  the 
technicians  consoles  when  on-station.  This  would  spread  the  number  of  areas  amongst  a 
greater  number  of  people  there  by  lowering  the  number  of  surveillance  areas  monitored 
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by  any  single  tracker.  The  other  option  is  to  simply  accept  a  lower  number  of  areas  to 
monitor  from  the  collection  manager.  By  showing  the  collection  manager  how  much 
better  they  do  when  not  having  to  monitor  so  many  areas  should  entice  the  collection 
manger  to  focus  their  efforts  to  receive  a  higher  value  of  information. 

The  next  recommendation  is  to  simply  fly  higher.  They  can  do  this  by  taking  on 
less  fuel  during  their  air  refuelings.  This  would  give  them  the  ability  to  fly  higher  for 
greater  amounts  of  their  on-station  time.  However,  taking  less  fuel  also  comes  at  price 
because  more  tankers  would  be  required  to  get  the  same  amount  of  coverage.  The  other 
issue  it  creates  is  more  time  will  be  spent  air  refueling  which  results  in  less  time  on- 
station.  The  tradeoff  here  comes  when  determining  if  it  is  more  important  to  have  more 
time  on-station  at  less  than  optimal  altitudes  or  is  it  better  to  forgo  some  on-station  time 
to  be  at  optimal  altitudes  for  longer  periods  of  time. 

Other  recommendations  are  to  create  relationships  with  outside  agencies  so  to 
increase  the  time  they  have  a  positive  identification  and  an  on  the  ground 
communications  asset.  By  creating  these  relationships,  they  will  also  help  with  fostering 
better  feedback  during  and  after  the  mission.  They  could  also  use  this  model  with  the 
Combined  Air  Operations  Center  (CAOC)  to  show  the  leaders  there  how  much  better 
there  information  would  be  if  provided  with  the  correct  cross-cue  assets.  The  final 
recommendations  are  to  track  in  low  areas  and  not  in  mountainous  areas  and  attempt  if  at 
all  possible  to  fly  in  dry  conditions.  It  was  shown  that  tracking  in  mountainous  areas 
provides  low  values  of  information.  It  would  be  much  better  to  reject  these  missions  and 
request  missions  where  the  probability  of  a  successful  mission  is  much  higher.  By  doing 
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some  or  all  of  these  things  it  is  possible  to  increase  the  value  of  a  mission  by  as  much  as 
30%. 

5.4  Verification  and  Validation 

The  model  verification  process  determines  if  the  model  meets  identified 
specification  and  ensures  the  model  is  doing  what  you  expect  it  is  doing  or  the 
mathematical  calculations  that  the  model  is  computing  are  indeed  correct.  To  verify  this 
model,  the  output  data  of  three  of  the  alternatives  were  compared  against  a  manual 
calculation  of  the  same  alternatives.  After  comparing  the  calculations,  it  was  noted  that 
the  manual  calculations  derived  the  same  scores  for  the  alternatives  as  the  model  which 
verified  that  the  model  was  working  in  the  manner  in  which  it  was  intended. 

The  model  validation  process  is  much  harder  and  is  a  way  of  evaluating  if  the 
model  meets  the  overall  project  objectives.  This  process  confirms  that  a  model  can 
effectively  be  applied  to  a  given  task.  This  particular  model  was  validated  in  two  ways. 
The  first  validation  took  place  when  comparing  the  data  the  MITRE  Corporation 
compiled  to  the  data  that  was  compiled  for  this  model  (Bonaceto,  Mooers,  Theophanis,  & 
Wrick,  2010).  Both  independent  studies  the  same  factors  to  be  important  when  modeling 
the  GMTI  process.  The  second  validation  process  came  from  the  crews  of  the  JSTARS 
themselves.  This  model  was  presented  to  crewmembers  other  than  the  ones  who  actually 
helped  build  the  model  to  determine  if  the  scores  of  the  current  missions  they  were  flying 
accurately  represented  their  mission  results.  The  crewmembers  felt  that  the  scores  there 
missions  were  receiving  was  extremely  close  to  the  mission  results  they  were  seeing  on 
their  flights. 
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5.5  Conclusion 


This  thesis  began  by  asking  how  can  the  effectiveness  of  a  GMTI  asset  be 
measured  when  there  is  no  magic  number  of  tanks,  boats,  cars  or  people  that  will  make 
this  capability  more  or  less  effective.  Understanding  that  intelligence  is  a  stochastic 
process,  determining  the  true  answer  to  this  question  is  probably  impossible.  By 
understanding  how  and  when  GMTI  and  JSTARS  have  been  effective  allowed  the 
JSTARS  intelligence  gathering  process  to  be  modeled  using  the  VFT  10  step  process. 
This  process  enabled  some  valuable  insights  to  be  gained  on  the  process  and  identified 
ways  to  optimize  the  use  of  the  GMTI  capability.  Specifically,  this  research  identified 
environments  or  situations  when  using  this  capability  will  and  will  not  provide  values  of 
information  that  are  adequate  enough  for  it  to  be  used.  Hopefully  DM  will  use  this  model 
to  make  better  decision  about  when  and  where  to  deploy  JSTARS.  This  template  of 
value-focused  thinking  can  be  used  to  determine  the  effectives  of  other  intelligence  assets 
in  the  DoD  arsenal. 
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Appendix  A:  JSTARS  Mission  Crew  Duties  &  Responsibilities 


Mission  Crew  Commander  (MCC). 

Responsible  authority  for  assigned  BM-C2ISR  mission  tasks  and  coordinates  with  the 
AC  to  ensure  effective  sortie  and  mission  accomplishment.  Supervise  execution  of  HHQ 
assigned  tasks.  Ensure  crewmember  adherence  to  Rules  of  Engagement  (ROE)  and 
SPINS.  During  decentralized  operations  the  MCC  is  the  onboard  authority  for 
determining  mission  tasking.  Declare  operations  normal/on-station/off-station  and  advise 
external  agencies  about  the  aircraft  status.  Collate  and  compile  mission  reports  and 
summaries.  Responsible  for  accounting  and  safeguarding  of  classified  materials  and 
proper  destruction.  Tailor  mission  crew  and  positional  responsibilities  based  upon 
mission  requirements  and  operations. 

Deputy  Mission  Crew  Commander  (DMCC). 

Act  as  Anny  liaison  to  MCC  and  mission  crew.  Ensure  that  the  Ground  Commander’s 
intent  is  understood  and  that  JSTARS  crewmembers  understand  how  ground  operations 
will  be  executed.  Ensure  the  ground  commander  and  common  ground  stations  (CGSs) 
are  aware  of  on-station/off-station  and  aircraft  status.  Manage  Information  flow  to 
supported  ground  units  via  radios  and  all  available  data  links  (FBCB2,  IDM,  SCDL, 
DATASAT,  &  AIRNET/INMARSAT).  Coordinate  with  the  ground  Fire  Support  Officer 
when  required. 

Airborne  Intelligence  Officer/Technician  (AOI/T). 

Analyze  incoming  reports  from  external  intelligence  collection  agencies  and  determine 
the  impact  on  mission  execution.  Ensures  amplifying  intelligence  data  is  fused  as 
applicable  to  enhance  the  BM-C2ISR  mission.  Verify  and  update  the  order  of  battle  data. 
Operation  of  the  Broadcast  Intelligence  system.  Report  radar  tracks  both 
intemally/extemally  to  intelligence  collection  agencies  for  further  collection  and 
amplification. 

Senior  Director  (SD). 

Monitor  and  assess  current  air/ground  situation;  coordinate  mission  changes  with 
appropriate  agencies.  Direct  BM-C2  mission  execution  with  regard  to  Find,  Fix,  Track, 
Target,  Engage  and  Assess  (F2T2EA).  Coordinate  with  the  SO  for  radar  management  and 
surveillance  operations.  C2  includes  procedural  control,  managing  mission  changes, 
striking  targets  and  directing  battlespace  logistical  efforts  (e.g.  tanker  flow).  Develop  an 
effective  communications  plan. 

Surveillance  Officer  (SO). 

Conduct  effective  radar  timeline  management;  inform  crew  of  sensor  anomalies. 
Coordinate  with  SD  for  management  of  the  Operations  Section.  The  SO  is  responsible  for 
signing  out  the  SO  Flyaway  Kit  from  DOW  and  carrying  on  every  mission  flight.  The  kit 
will  contain: 
l.T.O.  1E-8C-43-1-1-1 
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2.  AFTTP  3-1  .JSTARS 

3.  Appendix  H  (Classified  PHB) 

4.  Classified  In-Flight  Guide  (IFG) 

5.  E-8C  Security  Classification  Guide  (SCG) 

Senior  Surveillance  Manager  (SSM). 

Ensure  tracking  responsibilities/continuity  in  the  AOR.  Coordinate  with  the  CST  for 
JTIDS  link  operations.  Oversee  activities  of  Surveillance  Section. 

Air  Weapons  Officer  (AWO). 

Conduct  BM-C2  mission  execution  with  regard  to  F2T2EA  using  procedural  control, 
target  engagement,  TAC  (A),  managing  ATO/ACO  changes  and  directing  battle  space 
logistical  efforts. 

Air  Operations  Technician  (AOT). 

Use  sensor  data  for  accurate  tracking  in  assigned  AOR. 

Airborne  Target  Surveillance  Supervisor(ATSS). 

Maintain  voice  and  SCDL  contact  with  CGS  to  accomplish  ground  component 
commander  objectives;  process  radar  service  requests  as  required. 

Airborne  Radar  Technician  (ART). 

Initiates,  operates  and  maintains  radar  and  O&C  (computer)  systems.  Monitors  system 
status  and  troubleshoots  malfunctions  to  keep  systems  operational,  and  acts  as  primary 
fire  fighter  for  emergencies  involving  these  systems. 

Communications  Systems  Technician  (CST). 

Initiates,  operates  and  maintains  all  aircraft  communications  including  voice  and  data  link 
systems.  Monitors  system  status  and  troubleshoots  malfunctions  to  keep  systems 
operational,  and  acts  as  primary  fire  fighter  for  emergencies  involving  these  systems. 
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MEASURE  OF  EFFECTIVENESS  FOR  JSTARS 
GROUND  MOVING  TARGET  INDICATOR:  A 


Appendix  B:  Storyboard 
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Appendix  C:  Single  Dimensional  Value  Functions 


f - \ 

IPB 

1.00 

0.90 

0.80 

0.70 

0.60 - 1 

OJ 

|  0.50 

>  0.40 

0.30 
0.20 

0.10 - U 


No 

Yes 

n  Category 

0.00 

1.00 

Figure  25:  Intelligence  Preparation  of  the  Battle  Field  SDVF 


Figure  26:  Number  of  Areas  Tram  is  tracking  SDVF 
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Figure  27:  Altitude  SDVF 
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Figure  28:  Distance  I  SDVF 


97 


r 


Weather 


1.00 

0.90 

0.80 

0.70 

0.60 

<u 

i  0.50 

> 

0.40 

0.30 

0.20 

0.10 


0.00 


Ice/Snow 

Heavy  Prec 

Light  Prec 

Dry 

■  Category 

0.00 

0.45 

0.80 

1.00 

Figure  29:  Weather  aircraft  is  operating  in  SDVF 
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Figure  30:  Type  of  target  SDVF 
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Figure  31:  Number  of  Targets  SDVF 
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Figure  32:  Positive  Identification  SDVF 
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Figure  33:  Communications  SDVF 


Figure  34:  Feedback  SDVF 
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Figure  35:  Location  SDVF 
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Figure  36:  De-confliction  SDVF 
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Figure  37:  Target  Terrain  SDVF 


Appendix  D:  Names  and  Positions  of  Subject  Matter  Experts 
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Appendix  E:  8  Attributes  of  Intelligence 


Anticipatory 

Intelligence  must  anticipate  the  informational  needs  of  the  commander  and  joint 
force  staff  in  order  to  provide  a  solid  foundation  for  operational  planning  and 
decision  making.  Anticipating  the  joint  force’s  intelligence  needs  requires  the 
intelligence  staff  to  identify  and  fully  understand  the  command’s  current  and 
potential  missions,  the  commander’s  intent,  all  relevant  aspects  of  the  operational 
environment,  and  all  possible  friendly  and  adversary  CO  As. 

Timely 

Intelligence  must  be  available  when  the  commander  requires  it.  Timely  intelligence 
enables  the  commander  to  anticipate  events  in  the  operational  area.  This,  in  turn, 
enables  the  commander  to  time  operations  for  maximum  effectiveness  and  to  avoid 
being  surprised. 

Accurate 

Intelligence  must  be  factually  correct,  convey  an  appreciation  for  facts  and  the 
situation  as  it  actually  exists,  and  provide  the  best  possible  estimate  of  the  enemy 
situation  and  CO  As  based  on  sound  judgment  of  all  information  available.  The 
accuracy  of  intelligence  products  may  be  enhanced  by  placing  proportionally  greater 
emphasis  on  information  reported  by  the  most  reliable  sources.  Source  reliability 
should  be  evaluated  through  a  feedback  process  in  which  past  information  received 
from  a  source  is  compared  with  the  actual  “ground  truth”  (i.e.,  when  subsequent 
events,  reports,  or  knowledge  confirm  the  source’s  accuracy). 

Usable 

Intelligence  must  be  tailored  to  the  specific  needs  of  the  commander,  and  must  be 
provided  in  forms  suitable  for  immediate  comprehension.  The  commander  must  be 
able  to  quickly  apply  intelligence  to  the  task  at  hand.  Providing  useful  intelligence 
requires  the  producers  to  understand  the  circumstances  under  which  their  products 
are  used.  Commanders  operate  under  mission,  operational,  and  time  constraints  that 
will  shape  their  intelligence  requirements  and  determine  how  much  time  they  will 
have  to  study  the  intelligence  that  they  are  provided.  Commanders  may  not  have 
sufficient  time  to  analyze  intelligence  reports  that  are  excessively  complex  and 
difficult  to  comprehend.  The  “bottom  line”  must  be  up  front  and  easily 
understandable.  Oral  presentations  should  be  simple  and  to  the  point.  The 
use  of  approved  joint  terms  and  straightforward  presentation  methods  will  facilitate 
rapid  and  effective  application  of  intelligence  to  support  joint  operations. 

Complete 

Complete  intelligence  answers  the  commander’s  questions  about  the  adversary  to 
the  fullest  degree  possible.  It  also  tells  the  commander  what  remains  unknown.  To 
be  complete,  intelligence  must  identify  all  adversary  capabilities  that  may  impact 
mission  accomplishment  or  execution  of  the  joint  operation.  Complete  intelligence 
informs  the  commander  of  all  major  CO  As  that  are  available  to  the  adversary 
commander,  and  identifies  those  assessed  as  most  likely  or  most  dangerous.  The 
effort  to  produce  complete  intelligence  never  ceases.  While  providing  available 
intelligence  to  those  who  need  it  when  they  need  it,  the  intelligence  staff  must  give 
priority  to  the  commander’s  unsatisfied  critical  requirements.  Intelligence 
organizations  must  anticipate  and  be  ready  to  respond  to  the  existing  and  contingent 
intelligence  requirements  of  commanders  and  forces  at  all  levels  of  command. 

Relevant 

Intelligence  must  be  relevant  to  the  planning  and  execution  of  the  operation  at  hand. 

It  must  aid  the  commander  in  the  accomplishment  of  the  command’s  mission. 
Intelligence  must  contribute  to  the  commander’s  understanding  of  the  adversary,  but 
not  burden  the  commander  with  intelligence  that  is  of  minimal  or  no  importance  to 
the  current  mission.  It  must  help  the  commander  decide  how  to  accomplish  the 
assigned  mission  without  being  unduly  hindered  by  the  adversary.  Commanders 
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must  communicate  their  intent  and  their  operational  concept  to  the  intelligence  staff 
if  relevant  intelligence  is  to  be  produced.  Requirements  must  be  updated  and  refined 
as  the  friendly  mission  or  the  adversary  situation  changes. 

Objective 

For  intelligence  to  be  objective,  it  should  be  unbiased,  undistorted,  and  free  of 
prejudicial  judgments.  The  objective  analyst  must  remain  open-minded  to  all 
hypotheses  and  should  never  attempt  to  make  the  facts  fit  preconceptions  of  a 
situation  or  an  adversary.  In  particular,  intelligence  should  recognize  each  adversary 
as  unique,  and  should  avoid  mirror  imaging.  Red  teams  should  be  used  to  check 
analytical  judgments  by  ensuring  assumptions  about  the  adversary  are  valid  and 
intelligence  assessments  are  free  from  mirror  imaging  and  cultural  bias. 

Available 

Intelligence  must  be  readily  accessible  to  the  commander.  Availability  is  a  function 
of  not  only  timeliness  and  usability,  but  also  appropriate  security  classification, 
interoperability,  and  connectivity.  Intelligence  producers  must  strive  to  provide 
data  at  the  lowest  level  of  classification  and  least  restrictive  releasability  caveats, 
thereby  maximizing  the  consumers’  access,  while  ensuring  that  sources  of 
information  and  methods  of  collection  are  fully  protected. 
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Appendix  F :  Blue  Dart 


The  Department  of  Defense  (DoD)  dedicates  a  huge  amount  of  its  budget  and 
manpower  to  the  intelligence  gathering  process.  In  today’s  war  environments  where 
the  fighting  is  mostly  unconventional,  the  DoD  depends  on  their  intelligence 
gathering  platforms  more  than  ever  to  provide  timely  and  accurate  information. 
Unfortunately,  it  can  sometimes  be  very  difficult  to  almost  impossible  to  ascertain 
how  effective  any  particular  intelligence  gathering  asset  really  is.  A  problem  that  has 
plagued  intelligence  gathering  systems  for  years  is  that  their  overall  effectiveness  is 
determined  using  measures  of  performance  rather  than  measures  of  effectiveness. 
Measures  of  Performance  (MOPs)  describe  how  well  a  system  utilizes  resources,  but 
Measures  of  Effectiveness  (MOEs)  are  quantitative  measures  that  give  some  insight 
into  how  effectively  a  unit  is  performing.  For  example,  if  a  ISR  assets  is  scheduled  to 
fly  8  hours  and  flies  8  hours  an  MOP  would  consider  that  asset  100%  effective,  but 
the  true  effectiveness  could  and  usually  is  far  less. 

With  tighten  fiscal  restraints  the  DoD  is  now  under  scrutiny  to  find  ways  to  cut 
useless  and  redundant  equipment  and  systems.  They  have  to  justify  every  piece  of 
equipment  required  to  maintain  the  safety  and  security  of  the  nation.  With  such  tight 
constraints,  every  asset  has  to  prove  its  worthiness  or  face  possible  budget  cuts. 
Additionally,  decision  makers  require  the  most  accurate  information  possible  to  make 
decisions  on  deployment  and  procurement  considerations  of  intelligence  assets. 

One  of  the  systems  and  capabilities  that  is  under  budget  attacks  and  uses  MOPs  to 
measure  its  overall  effectiveness  is  the  Joint  Surveillance  Target  Attack  Radar  System 
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or  JSTARS  with  its  Ground  Moving  Target  Indicator  (GMTI)  system.  The  difficulty 
with  the  system  is  that  it  can  see  one  hundred  or  one  million  vehicles,  but  their  overall 
effectiveness  does  not  increase  or  decrease  based  on  the  number  of  vehicles  seen. 
History  shows  that  there  have  been  situations  where  JSTARS  has  been  extremely 
successful  and  others  where  the  platform  has  struggled  to  have  an  impact  on  the 
conflict.  Using  this  information  allows  MOEs  to  be  created  which  give  much  better 
insight  into  how  effective  the  JSTARS  asset  can  and  will  be  in  the  future. 

Understanding  the  complexities  and  constraints  of  the  system  using  the  Decision 
Analysis  discipline,  the  Air  Force  Institute  of  Technology  has  created  a  value-focused 
thinking  model  which  models  the  JSTARS  intelligence  gathering  process.  The  model 
is  built  in  a  hierarchal  structure  and  identifies  values  and  measures  that  are  important 
to  the  GMTI  process  onboard  JSTARS  and  assign  weights  to  those  values  and 
measures.  Single  dimensional  value  functions  are  then  assigned  to  each  measure 
which  allows  a  score  to  be  assigned  to  every  possible  scenario  or  environment  that  the 
assets  could  possibly  enter.  Using  the  assigned  scores  and  sensitivity  analysis  allows 
the  user  to  identify  scenarios  where  the  asset/capability  will  be  extremely  effective 
and  when  it  would  be  better  not  to  use  the  system  as  the  value  of  information 
provided  by  the  system  will  be  extremely  low. 

The  model  helps  identify  key  controllable  and  uncontrollable  factors  that  affect 
the  system  and  the  ones  that  should  be  addressed  first  to  increase  mission  values.  By 
using  this  model,  decision  makers  will  have  the  ability  to  make  better  decisions  on 
what  theaters  will  be  most  applicable  to  using  the  JSTARS  GMTI  capability.  They 
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will  also  have  better  insight  on  what  upgrades  or  equipment  would  allow  the  aircraft 
perform  its  mission  more  successfully.  The  data  will  also  help  planners  better 
understand  where  and  how  to  employ  the  asset  to  maximize  its  effectiveness. 
Furthermore,  crews  will  have  insight  on  the  factors  that  will  affect  their  mission  prior 
to  ever  being  deployed  into  theater.  They  can  then  focus  their  efforts  on  the 
controllable  variables  such  as  feedback  or  having  a  positive  identification  asset  in  an 
effort  to  increases  the  overall  mission  scores.  Finally,  this  research  will  help  other 
modelers  better  model  GMTI  and  thus  make  more  accurate  assumptions  about  how 
many  GMTI  assets  are  required  and  what  can  actually  be  seen  using  a  GMTI  asset. 


107 


Bibliography 


AFDD,  2.-9.  (2007).  Intelliegnce,  Surveillance  and  Reconnaissance  Operations  .  HQ 

AFDC/DD. 

Albers,  T.  P.  (2001).  Joint  Surveillance  Target  Attack  Radar  System:  Unlimited 

Poetntial— Limited  Resourses.  Fort  Leaveworth:  US  Army  Command  and  General 
Staff  College. 

Antony,  P.,  Dunn,  J.,  Farr,  W.,  Rhodes,  D.,  Roedler,  G.,  Tilton,  C.,  et  al.  (1998).  Systems 
Engineering  Measurement  Primer.  San  Diego:  INCOSE. 

Arquilla,  J.  (2010,  March/ April).  The  New  Rules  of  War.  Foreign  Policy  ,  pp.  60-68. 

Bonaceto,  C.,  Mooers,  E.,  Theophanis,  S.,  &  Wrick,  V.  (2010,  August  13).  GMTI 
Resource  Management.  Command  &  Control  Center  .  McLean,  VA,  USA:  MITRE 
Corporation. 

Bullock,  R.  K.  (2006).  Theroy  of  Effectiveness  Measurement.  Wright  Patterson  AFB:  Air 
Force  Institute  of  Technology. 

Bunn,  D.  (1984).  Applied  Decision  Analysis.  New  York:  McGraw-Hill. 

Clemen,  R.  T.,  &  Reilly,  T.  (2001).  Making  Hard  Decisions  with  Decisions  Tools. 
Duxbury. 

Clevenger,  D.  R.  (1996).  "Battle  of  Khafji”  Air  Power  Effectiveness  In  The  Desert.  Air 
Force  Studies  and  Analyses  Agency  Force  Application  Division. 

Cote,  M.  D.  (2010).  Screening  and  Sufficiency  in  Multiobjective  Decision  Problems  with 
Large  Alternative  Sets.  Wright  Patterson  AFB:  Air  Force  Institute  of  Technology. 

Defense,  D.  o.  (2007,  December  12).  Basic  Tools  for  Process  Improvement:  Module  4 
Affinity  Diagram.  Retrieved  January  14,  2011,  from  Acqusition  Community 
Connection:  https://acc.dau.mil/adl/en-US/184574/file/32039/opdef.pdf 

Defense,  D.  o.  (2007).  Joint  Intelligence,  JP  2.0.  Washington  DC. 

Dunn,  R.  J.,  Bingham,  P.  T.,  &  Fowler,  C.  A.  (2004).  Ground  Moving  Target  Indicator 
Radar  and  the  Transformation  of  US  Warfighting.  Northrop  Grumman. 

Eccles,  R.  G.  (1991).  The  Performance  Measurement  Manifesto.  Harvard  Business 
Reveiw  ,  131-137. 


108 


Farringdon,  H.  (n.d.).  Retrieved  November  13,  2010 

Gartner,  S.  S.  (1997).  Strategic  Assessment  in  War.  New  Haven:  Yale  University. 

Gates,  R.  M.  (2009,  July  16).  US  Department  of  Defense.  Retrieved  Jan  4,  2011,  from 
Defense.Gov:  http://www.defense. gov/transcripts/transcript. aspx?transcriptid=4445 

Gates,  R.  M.  (2010,  August  9).  US  Department  of  Defense.  Retrieved  November  14, 

2010,  from  http://www.defense.gov 

Gawande,  K.,  &  Wheeler,  T.  (1999).  Measures  of  Effectiveness  for  Governmental 
Organizations.  Management  Science  ,  42-58. 

Geisler,  E.  (2000).  The  Metrics  of  Science  and  Technology.  Westport,  CT:  Quorum 
Books. 

Jordan,  G.,  Prevette,  S.,  &  Woodward,  S.  (2001).  The  Performance-Based  Management 
Handbook:  Analyzing,  Reviewing  and  Reporting  Performance  Data.  Training 
Resources  and  Data  Exchange  Performance-Based  Management  Special  Interest 
Group. 

Jurk,  D.  M.  (2002).  Decision  analysis  with  value  focused  thinking  as  a  methodology  to 
select  force  protection.  Thesis,  Air  Force  Institute  of  Technology,  Graduate  School  of 
Engineering  and  Management. 

Kaplan,  R.  S.,  &  Norton,  D.  P.  (1996).  Using  the  Balanced  Scorecard  as  a  Strategic 
Management  System.  Harard  Business  Review  ,  1-13. 

Keeney,  R.  L.  (1992).  Value  Focused  Thinking:  A  Path  to  Creative  Decision  Making. 
Cambridge:  Harvard  University  Press. 

Kirkwood,  C.  W.  (1997).  Strategic  Decsision  Making.  Wadsworth:  Cengage  Learning. 

Marston,  R.  (2009,  May).  Behappy!  Motivitional  Quotes.  Retrieved  January  7,  2011, 
from  Behappy  101:  http://www. behappy  1 0 1  .com/motivational-quotes .html 

Melenyk,  S.  A.,  Stewart,  D.  M.,  &  Swink,  M.  (2004).  Metrics  and  Performance 

Measurement  in  Ooperations  Management:  Dealing  with  the  Metrics  Maze.  Journal 
of  Operations  Management ,  209-217. 

Merrian-Webster.  (2011,  January  6).  Intelliegnce.  Retrieved  January  6,  2011,  from 
http://www.merriam-webster.com/dictionary/intelligence 


109 


Park,  R.  E.,  Goethert,  W.  B.,  &  Florae,  W.  A.  (1996).  Goal-Driven  Software 
Measurement-A  Guidebook.  Pittsburg:  Carnegie  Mellon  University. 

Pfanzagl,  J.  (1971).  Theroy  of  Measurement.  Wurzburg:  Physica-Verlag. 

Pruitt,  K.  A.  (2003).  Modeling  Homeland  Security:  A  Vcdue  Focused  Thinking  Approach. 
Wright  Patterson,  AFB:  Air  Force  Institute  of  Technology. 

Sahadi,  J.  (2010,  July  9).  Defense  Spending:  Slaying  the  sacred  cow.  Retrieved  Jan  4, 
2011,  from 

http://money.cnn.com/2010/07/09/news/economy/defense_spending/index.htm 

Shoviak,  M.  J.  (2001).  Decision  Analysis  Methodology  to  Evaluate  Intergrated  Solid 
Waste  Management  Alternatives  for  a  Remote  Alaskan  Air  Station.  Wright  Patterson, 
AFB:  Air  Force  Institute  of  Technology. 

Sproles,  N.  (1997).  Meaning,  Mapping,  Measures,  and  Test  &  Evaluation.  ITEA  journal 
of  test  and  evaluation  vol.  17,  no.  3,  pp.  34-39 , 16-31. 

Stewart,  B.  G.  (1991).  Ooperation  Desert  Storm  The  Military  Intelliegence  Story.  US 
Army. 

Vol  3,  A.  1.-2. -8.  (2009).  E-8  Ooperations  Procedures.  Fangley:  HQ  ACC/A3YA. 

Weir,  J.  D.  (2010,  April  6).  Notes  from  OPER  643  Advanded  Decision  Analysis.  OPER 
643  Week  2  .  Wright  Patterson  AFB,  OH,  United  States:  Air  Force  Institue  of 
Tecnology. 


no 


Vita 


Major  Gardner  Jerell  Joyner  graduated  from  Fuquay  Varina  High  School  in 
Fuquay  Varina,  North  Carolina.  He  entered  undergraduate  studies  at  the  Fayetteville 
State  University,  Fayetteville  North  Carolina  where  he  graduated  with  a  Bachelor  of 
Science  degree  in  Mathematics  Teaching  in  May  1997.  He  was  commissioned  through 
the  Detachment  607  AFROTC  at  Fayetteville  State  University  where  he  was  chosen  to  be 
an  aviator  in  the  US  Air  Force. 

His  first  assignment  was  at  Pensacola  Florida  as  a  student  in  Undergraduate 
Navigator  Training  in  September  1997.  He  completed  his  training  In  Dec  1998  and  was 
assigned  to  the  91 1th  Air  Refueling  Squadron,  Grand  Forks,  North  Dakota  where  he  was  a 
navigator  and  squadron  scheduler.  While  stationed  at  Grand  Forks  he  deployed  often  to 
Operations  Northern  and  Southern  Watch.  He  then  PCS’ed  to  Warner  Robins  AFB, 
where  he  was  a  member  of  the  1 16  ACW.  While  there  he  deployed  multiple  times  to 
Operations  ENDURING  FREEDOM  and  IRAQI  FREEDOM.  After  completion  of  this 
assignment  he  moved  on  to  be  a  part  of  the  552d  ACW.  There  he  was  a  flight  commander 
and  the  chief  of  flight  deck  training  for  the  wing  before  moving  on  o  his  next  assignment 
in  San  Antonio,  Texas.  In  San  Antonio,  he  was  assigned  to  the  Air  Force  Personnel 
Center  where  he  was  responsible  for  the  assignments  of  all  C2ISR  pilots  and  navigators. 
While  there  he  also  had  the  pleasure  of  standing  up  the  Air  Force’s  newest  weapon 
system,  the  MC-12.  In  August  2009,  he  entered  the  Graduate  School  of  Engineering  and 


111 


Management,  Air  Force  Institute  of  Technology.  Upon  graduation,  he  will  be  assigned  to 
the  Robins  AFB,  GA. 


112 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  074-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of 
information,  including  suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188), 

1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  an 
penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1 .  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE 

24-03-2011  Master’s  Thesis 

3.  DATES  COVERED  (From  -  To) 

Sep  2010 -Mar  2011 

4.  TITLE  AND  SUBTITLE 

MEASURE  OF  EFFECTIVENESS  FOR  JSTARS  GROUND  MOVING  TARGET 
INDICATOR:  A  VALUE  FOCUSED  THINKING  APPROACH 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Joyner,  Gardner  J.,  Major,  USAF 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

Graduate  School  of  Engineering  and  Management  (AFIT/EN) 

2950  Hobson  Street,  Building  642 

WPAFB  OH  45433-7765 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

AFIT -OR-MS  -EN  S  - 1 1  - 1 1 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Defense  Intelligence  Agency 

Office  of  Assessments 

JFCC-ISR  DJS-5,  Bldg  6000 

200  MacDill  Blvd 

Boling  AFB,  Washington  DC  20340 

POC:  Paul  Plescow  (202)  231-6574,  Paul.Plescow@dia.mil 

10.  SPONSOR/MONITOR’S  ACRONYM (S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABIUTY  STATEMENT 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

As  the  nature  of  warfare  has  shifted  from  a  conventional  approach  to  more  guerilla  type  warfare,  intelligence  has  become  more  important  than  at  any 
other  time  in  the  history  of  the  United  States  Military.  With  the  stochastic  nature  of  intelligence  gathering,  it  is  almost  impossible  to  know  with  any 
degree  of  certainty  where  and  when  the  next  piece  of  information  that  could  possibly  change  the  course  of  the  battle  or  war  will  be  obtained.  US 
intelligence  gathering  assets  have  long  been  plagued  with  using  useless  measures  of  performance  rather  than  measures  of  effectiveness  to  determine 
their  worth.  This  research  uses  a  value  focused  thinking  approach  to  determine  the  effectiveness  of  a  specific  capability  or  asset.  Specifically,  it  looks 

at  Ground  Moving  Target  Indicator  onboard  the  E-8C  Joint  Surveillance  Target  Attack  Radar  System.  This  research  attempts  to  provide  a  model  to  a 
decision  maker  so  he  or  she  will  know  in  advance  the  approximate  value  of  information  they  will  receive  from  a  particular  asset  or  capability  before 
the  asset  is  ever  deployed  into  the  area  of  responsibility. 

15.  SUBJECT  TERMS 

Joint  Surveillance  Target  Attack  Radar  System,  Ground  Moving  Target  Indicator,  Measures  of  Effectiveness,  Value-Focused  Thinking 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

1 9a.  NAME  OF  RESPONSIBLE  PERSON 

ABSTRACT 

OF 

Jeffery  L.  Weir,  PHD 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

PAGES 

19b.  TELEPHONE  NUM BER  (Include  area  code) 

u 

u 

u 

UU 

127 

(937)  255-6565,  ext  4314;  e-mail:  Jeffery.Weir@afit.edu 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39-18 


